2024-07-26T05:59:23.321 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T05:59:23.343 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T05:59:23.443 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960 branch: rishabh-mgr-vol-clone-stats-bugfixes description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} email: null first_in_suite: false job_id: '7818960' kernel: client: branch: testing flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-mgr-vol-clone-stats-bugfixes ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 4aae82f07003420ccc152fe0b01f3d5a44885708 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4aae82f07003420ccc152fe0b01f3d5a44885708 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-mgr-vol-clone-stats-bugfixes sha1: 4aae82f07003420ccc152fe0b01f3d5a44885708 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6921 sha1: 4aae82f07003420ccc152fe0b01f3d5a44885708 sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-mgr-vol-clone-stats-bugfixes suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 4aae82f07003420ccc152fe0b01f3d5a44885708 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJPgve1quZVfhUSvWDNm7uSDkV8/n1fQNdG2XGx19Q/L0XaeVA4kQADjwEAzvpiP+Kn8Lh4ySQlH+QmZfHFBTJY= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC+emzptfrRxxWzFEY3+r2b+TICzh8x5GxHKRxKsmq39jz8M93Os7XJC7FgvlB2CPfwqb+gk0lAwQVLgXuY1RnY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestMisc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_05:39:42 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T05:59:23.444 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa; will attempt to use it 2024-07-26T05:59:23.444 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks 2024-07-26T05:59:23.445 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T05:59:23.446 INFO:teuthology.task.internal:Checking packages... 2024-07-26T05:59:23.467 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4aae82f07003420ccc152fe0b01f3d5a44885708' 2024-07-26T05:59:23.467 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T05:59:23.467 INFO:teuthology.packaging:ref: None 2024-07-26T05:59:23.467 INFO:teuthology.packaging:tag: None 2024-07-26T05:59:23.467 INFO:teuthology.packaging:branch: rishabh-mgr-vol-clone-stats-bugfixes 2024-07-26T05:59:23.468 INFO:teuthology.packaging:sha1: 4aae82f07003420ccc152fe0b01f3d5a44885708 2024-07-26T05:59:23.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-mgr-vol-clone-stats-bugfixes 2024-07-26T05:59:23.611 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4588.g4aae82f0 2024-07-26T05:59:23.613 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T05:59:23.620 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T05:59:23.620 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T05:59:23.677 INFO:teuthology.task.internal:Saving configuration 2024-07-26T05:59:23.696 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T05:59:23.791 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T05:59:23.815 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 05:53:50.561379', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJPgve1quZVfhUSvWDNm7uSDkV8/n1fQNdG2XGx19Q/L0XaeVA4kQADjwEAzvpiP+Kn8Lh4ySQlH+QmZfHFBTJY='} 2024-07-26T05:59:23.836 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 05:53:50.559313', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC+emzptfrRxxWzFEY3+r2b+TICzh8x5GxHKRxKsmq39jz8M93Os7XJC7FgvlB2CPfwqb+gk0lAwQVLgXuY1RnY='} 2024-07-26T05:59:23.836 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T05:59:23.843 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-07-26T05:59:23.843 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-07-26T05:59:23.843 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T05:59:23.981 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f258dbac0>, signals=[15]) 2024-07-26T05:59:23.981 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T05:59:23.987 INFO:teuthology.task.internal:Opening connections... 2024-07-26T05:59:23.988 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T05:59:23.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T05:59:24.067 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T05:59:24.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T05:59:24.140 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T05:59:24.147 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-07-26T05:59:24.164 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-07-26T05:59:24.165 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-07-26T05:59:24.222 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T05:59:24.223 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-07-26T05:59:24.223 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T05:59:24.223 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T05:59:24.223 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T05:59:24.223 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-07-26T05:59:24.248 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-07-26T05:59:24.266 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-07-26T05:59:24.266 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-07-26T05:59:24.323 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-07-26T05:59:24.323 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-07-26T05:59:24.324 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T05:59:24.325 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T05:59:24.325 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T05:59:24.325 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-07-26T05:59:24.348 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T05:59:24.357 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T05:59:24.373 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T05:59:24.373 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-07-26T05:59:24.376 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-07-26T05:59:24.393 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T05:59:24.399 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T05:59:24.399 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-07-26T05:59:24.433 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-07-26T05:59:24.730 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T05:59:24.779 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T05:59:24.799 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T05:59:24.800 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-07-26T05:59:24.800 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-07-26T05:59:24.800 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T05:59:24.800 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T05:59:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T05:59:24.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-07-26T05:59:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T05:59:24.821 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-07-26T05:59:25.006 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T05:59:25.007 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T05:59:25.007 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-07-26T05:59:25.008 DEBUG:teuthology.task.kernel:sha1 for client.1 is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T05:59:25.008 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T05:59:25.008 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-07-26T05:59:25.025 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-437.el9.x86_64 2024-07-26T05:59:25.025 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T05:59:25.026 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T05:59:25.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum upgrade -y linux-firmware 2024-07-26T05:59:25.027 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-437.el9.x86_64 2024-07-26T05:59:25.028 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T05:59:25.028 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T05:59:25.028 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum upgrade -y linux-firmware 2024-07-26T05:59:28.607 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-07-26T05:59:29.516 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.2 MB 00:02 2024-07-26T05:59:30.683 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-07-26T05:59:31.217 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-07-26T05:59:35.826 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.5 MB 00:00 2024-07-26T05:59:36.383 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-07-26T05:59:38.101 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 18 kB 00:01 2024-07-26T05:59:38.559 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 18 kB 00:00 2024-07-26T05:59:38.914 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 33 MB/s | 22 MB 00:00 2024-07-26T05:59:39.679 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:00 2024-07-26T05:59:44.161 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-07-26T05:59:45.101 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-07-26T05:59:46.278 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-07-26T05:59:46.293 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-07-26T05:59:46.293 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-07-26T05:59:46.293 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-07-26T05:59:46.293 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-07-26T05:59:46.293 INFO:teuthology.orchestra.run.smithi164.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T05:59:46.293 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T05:59:46.294 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-07-26T05:59:46.294 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-07-26T05:59:46.294 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 1 Package 2024-07-26T05:59:46.294 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T05:59:46.294 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 383 M 2024-07-26T05:59:46.294 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-07-26T05:59:47.215 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-07-26T05:59:47.222 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-07-26T05:59:47.222 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 1 Package 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 383 M 2024-07-26T05:59:47.223 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-07-26T05:59:55.304 INFO:teuthology.orchestra.run.smithi164.stdout:linux-firmware-20240716-145.el9.noarch.rpm 43 MB/s | 383 MB 00:08 2024-07-26T05:59:55.304 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-07-26T05:59:55.304 INFO:teuthology.orchestra.run.smithi164.stdout:Total 43 MB/s | 383 MB 00:09 2024-07-26T05:59:57.602 INFO:teuthology.orchestra.run.smithi163.stdout:linux-firmware-20240716-145.el9.noarch.rpm 37 MB/s | 383 MB 00:10 2024-07-26T05:59:57.603 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-07-26T05:59:57.603 INFO:teuthology.orchestra.run.smithi163.stdout:Total 37 MB/s | 383 MB 00:10 2024-07-26T05:59:58.393 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-07-26T05:59:58.490 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-07-26T05:59:58.490 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-07-26T06:00:00.697 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-07-26T06:00:00.844 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-07-26T06:00:00.844 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-07-26T06:00:01.756 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-07-26T06:00:01.757 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-07-26T06:00:04.120 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-07-26T06:00:04.120 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-07-26T06:00:11.461 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-07-26T06:00:11.880 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T06:00:11.967 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T06:00:12.653 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T06:00:12.653 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T06:00:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T06:00:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:00:13.380 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-07-26T06:00:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T06:00:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:00:13.380 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-07-26T06:00:13.671 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.1... 2024-07-26T06:00:13.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T06:00:13.677 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-07-26T06:00:13.829 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T06:00:13.829 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T06:00:13.832 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi164.front.sepia.ceph.com to complete... 2024-07-26T06:00:14.388 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T06:00:14.468 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T06:00:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T06:00:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T06:00:15.901 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T06:00:15.901 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:00:15.901 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-07-26T06:00:15.901 INFO:teuthology.orchestra.run.smithi163.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T06:00:15.901 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:00:15.901 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-07-26T06:00:16.223 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T06:00:16.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T06:00:16.373 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T06:00:16.373 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T06:00:16.375 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi163.front.sepia.ceph.com to complete... 2024-07-26T06:00:22.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26 06:00:22 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T06:00:22.620 INFO:teuthology.orchestra.run.smithi164.stderr:FINISHED --2024-07-26 06:00:22-- 2024-07-26T06:00:22.620 INFO:teuthology.orchestra.run.smithi164.stderr:Total wall clock time: 8.8s 2024-07-26T06:00:22.620 INFO:teuthology.orchestra.run.smithi164.stderr:Downloaded: 1 files, 342M in 8.5s (40.2 MB/s) 2024-07-26T06:00:22.625 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.1... 2024-07-26T06:00:22.625 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T06:00:24.682 INFO:teuthology.orchestra.run.smithi164.stdout:Verifying... ######################################## 2024-07-26T06:00:24.789 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing... ######################################## 2024-07-26T06:00:24.839 INFO:teuthology.orchestra.run.smithi164.stdout:Updating / installing... 2024-07-26T06:00:25.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26 06:00:25 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T06:00:25.970 INFO:teuthology.orchestra.run.smithi163.stderr:FINISHED --2024-07-26 06:00:25-- 2024-07-26T06:00:25.970 INFO:teuthology.orchestra.run.smithi163.stderr:Total wall clock time: 9.6s 2024-07-26T06:00:25.970 INFO:teuthology.orchestra.run.smithi163.stderr:Downloaded: 1 files, 342M in 9.2s (37.3 MB/s) 2024-07-26T06:00:25.974 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T06:00:25.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T06:00:28.021 INFO:teuthology.orchestra.run.smithi163.stdout:Verifying... ######################################## 2024-07-26T06:00:28.073 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing... ######################################## 2024-07-26T06:00:28.123 INFO:teuthology.orchestra.run.smithi163.stdout:Updating / installing... 2024-07-26T06:00:32.884 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T06:00:36.133 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T06:01:12.570 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T06:01:12.571 DEBUG:teuthology.orchestra.run.smithi164:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T06:01:12.639 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T06:01:12.640 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T06:01:12.641 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T06:01:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T06:01:12.643 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T06:01:12.644 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T06:01:12.645 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T06:01:12.646 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T06:01:12.647 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T06:01:12.648 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T06:01:12.649 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T06:01:12.650 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T06:01:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T06:01:12.652 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T06:01:12.653 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T06:01:12.654 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T06:01:12.655 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T06:01:12.656 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T06:01:12.657 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T06:01:12.658 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T06:01:12.659 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T06:01:12.660 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T06:01:12.661 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T06:01:12.662 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T06:01:12.663 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T06:01:12.664 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T06:01:12.665 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T06:01:12.666 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T06:01:12.667 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T06:01:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T06:01:12.669 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T06:01:12.670 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T06:01:12.671 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T06:01:12.672 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T06:01:12.673 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T06:01:12.674 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T06:01:12.676 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T06:01:12.677 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T06:01:12.678 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T06:01:12.679 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T06:01:12.680 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T06:01:12.681 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T06:01:12.682 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T06:01:12.683 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T06:01:12.684 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T06:01:12.685 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T06:01:12.686 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T06:01:12.687 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T06:01:12.688 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T06:01:12.689 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T06:01:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T06:01:12.691 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T06:01:12.692 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T06:01:12.693 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T06:01:12.694 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T06:01:12.695 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T06:01:12.696 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T06:01:12.697 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T06:01:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T06:01:12.699 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T06:01:12.700 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T06:01:12.701 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T06:01:12.702 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T06:01:12.703 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T06:01:12.704 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T06:01:12.705 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T06:01:12.706 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T06:01:12.707 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T06:01:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T06:01:12.709 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T06:01:12.710 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T06:01:12.711 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T06:01:12.712 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T06:01:12.713 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T06:01:12.714 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T06:01:12.715 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T06:01:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T06:01:12.717 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T06:01:12.718 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T06:01:12.719 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T06:01:12.720 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T06:01:12.721 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T06:01:12.722 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T06:01:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T06:01:12.724 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T06:01:12.725 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T06:01:12.726 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T06:01:12.727 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T06:01:12.728 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T06:01:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T06:01:12.730 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T06:01:12.731 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T06:01:12.732 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T06:01:12.733 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T06:01:12.734 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T06:01:12.735 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T06:01:12.736 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T06:01:12.737 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T06:01:12.738 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T06:01:12.739 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T06:01:12.740 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T06:01:12.741 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T06:01:12.742 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T06:01:12.743 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T06:01:12.744 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T06:01:12.745 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T06:01:12.746 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T06:01:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T06:01:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T06:01:12.749 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T06:01:12.750 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T06:01:12.751 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T06:01:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T06:01:12.753 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T06:01:12.754 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T06:01:12.755 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T06:01:12.756 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T06:01:12.757 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T06:01:12.758 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T06:01:12.759 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T06:01:12.760 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T06:01:12.761 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T06:01:12.762 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T06:01:12.763 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T06:01:12.764 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T06:01:12.765 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T06:01:12.766 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T06:01:12.767 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T06:01:12.768 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T06:01:12.769 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T06:01:12.770 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T06:01:12.771 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T06:01:12.772 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T06:01:12.773 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T06:01:12.774 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T06:01:12.775 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T06:01:12.776 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T06:01:12.777 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T06:01:12.778 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T06:01:12.779 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T06:01:12.780 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T06:01:12.781 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T06:01:12.782 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T06:01:12.783 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T06:01:12.784 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T06:01:12.785 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T06:01:12.786 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T06:01:12.787 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T06:01:12.788 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T06:01:12.789 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T06:01:12.789 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T06:01:12.789 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T06:01:12.789 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T06:01:12.793 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T06:01:12.793 DEBUG:teuthology.orchestra.run.smithi164:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout:for file in vmlinuz System.map config; do 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T06:01:12.816 INFO:teuthology.orchestra.run.smithi164.stdout: fi 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:done 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T06:01:12.817 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2024-07-26T06:01:12.818 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 68.el9 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8264254 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T06:01:12.900 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:01:12.901 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T06:01:12.902 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T06:01:12.903 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 6.10.0-g3e9255e90710 2024-07-26T06:01:12.903 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T06:01:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-07-26T06:01:14.834 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T06:01:14.861 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-07-26T06:01:14.864 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-07-26T06:01:14.878 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.Av7tNVQ131 2024-07-26T06:01:14.879 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Av7tNVQ131 2024-07-26T06:01:14.943 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.Av7tNVQ131 2024-07-26T06:01:15.080 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.Av7tNVQ131 is 6KB 2024-07-26T06:01:15.129 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.Av7tNVQ131 2024-07-26T06:01:15.144 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T06:01:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T06:01:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T06:01:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T06:01:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T06:01:15.211 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T06:01:15.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-07-26T06:01:16.566 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T06:01:16.567 DEBUG:teuthology.orchestra.run.smithi163:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T06:01:16.646 INFO:teuthology.orchestra.run.smithi163.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T06:01:16.646 INFO:teuthology.orchestra.run.smithi163.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T06:01:16.646 INFO:teuthology.orchestra.run.smithi163.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T06:01:16.646 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T06:01:16.646 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T06:01:16.647 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T06:01:16.648 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T06:01:16.649 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T06:01:16.650 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T06:01:16.651 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T06:01:16.652 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T06:01:16.652 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T06:01:16.652 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T06:01:16.652 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T06:01:16.652 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T06:01:16.652 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T06:01:16.653 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T06:01:16.653 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T06:01:16.653 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T06:01:16.653 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T06:01:16.653 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T06:01:16.653 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T06:01:16.654 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T06:01:16.655 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T06:01:16.656 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T06:01:16.657 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T06:01:16.658 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T06:01:16.659 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T06:01:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T06:01:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T06:01:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T06:01:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T06:01:16.664 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T06:01:16.665 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T06:01:16.666 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T06:01:16.667 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T06:01:16.668 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T06:01:16.669 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T06:01:16.670 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T06:01:16.671 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T06:01:16.672 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T06:01:16.673 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T06:01:16.674 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T06:01:16.675 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T06:01:16.676 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T06:01:16.677 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T06:01:16.678 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T06:01:16.679 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T06:01:16.680 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T06:01:16.681 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T06:01:16.682 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T06:01:16.683 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T06:01:16.684 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T06:01:16.685 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T06:01:16.686 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T06:01:16.687 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T06:01:16.688 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T06:01:16.688 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T06:01:16.688 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T06:01:16.688 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T06:01:16.688 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T06:01:16.688 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T06:01:16.689 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T06:01:16.689 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T06:01:16.689 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T06:01:16.689 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T06:01:16.689 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T06:01:16.690 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T06:01:16.691 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T06:01:16.692 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T06:01:16.692 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T06:01:16.692 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T06:01:16.692 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T06:01:16.692 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T06:01:16.692 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T06:01:16.693 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T06:01:16.694 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T06:01:16.694 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T06:01:16.694 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T06:01:16.694 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T06:01:16.694 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T06:01:16.694 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T06:01:16.695 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T06:01:16.695 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T06:01:16.695 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T06:01:16.695 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T06:01:16.695 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T06:01:16.696 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T06:01:16.697 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T06:01:16.698 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T06:01:16.699 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T06:01:16.700 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T06:01:16.701 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T06:01:16.702 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T06:01:16.703 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T06:01:16.704 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T06:01:16.705 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T06:01:16.706 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T06:01:16.707 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T06:01:16.708 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T06:01:16.709 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T06:01:16.710 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T06:01:16.711 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T06:01:16.712 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T06:01:16.713 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T06:01:16.714 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T06:01:16.715 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T06:01:16.716 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T06:01:16.717 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T06:01:16.718 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T06:01:16.719 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T06:01:16.720 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T06:01:16.721 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T06:01:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T06:01:16.723 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T06:01:16.724 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T06:01:16.725 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T06:01:16.726 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T06:01:16.727 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T06:01:16.728 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T06:01:16.729 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T06:01:16.730 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T06:01:16.731 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T06:01:16.732 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T06:01:16.733 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T06:01:16.734 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T06:01:16.735 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T06:01:16.736 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T06:01:16.737 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T06:01:16.738 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T06:01:16.739 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T06:01:16.740 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T06:01:16.741 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T06:01:16.742 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T06:01:16.743 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T06:01:16.743 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T06:01:16.744 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T06:01:16.745 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T06:01:16.746 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T06:01:16.747 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T06:01:16.748 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T06:01:16.749 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T06:01:16.750 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T06:01:16.751 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T06:01:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T06:01:16.753 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T06:01:16.754 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T06:01:16.755 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T06:01:16.756 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T06:01:16.757 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T06:01:16.758 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T06:01:16.759 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T06:01:16.760 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T06:01:16.761 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T06:01:16.762 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T06:01:16.763 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T06:01:16.764 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T06:01:16.765 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T06:01:16.766 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T06:01:16.767 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T06:01:16.768 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T06:01:16.769 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T06:01:16.770 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T06:01:16.771 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T06:01:16.772 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T06:01:16.773 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T06:01:16.774 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T06:01:16.775 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T06:01:16.776 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T06:01:16.777 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T06:01:16.778 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T06:01:16.779 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T06:01:16.780 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T06:01:16.781 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T06:01:16.782 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T06:01:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T06:01:16.784 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T06:01:16.785 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T06:01:16.786 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T06:01:16.787 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T06:01:16.788 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T06:01:16.789 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T06:01:16.790 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T06:01:16.791 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T06:01:16.792 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T06:01:16.793 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T06:01:16.794 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T06:01:16.795 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T06:01:16.796 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T06:01:16.797 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T06:01:16.798 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T06:01:16.799 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T06:01:16.800 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T06:01:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T06:01:16.802 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T06:01:16.803 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T06:01:16.804 INFO:teuthology.orchestra.run.smithi163.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T06:01:16.808 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T06:01:16.808 DEBUG:teuthology.orchestra.run.smithi163:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout:fi 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout:for file in vmlinuz System.map config; do 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T06:01:16.833 INFO:teuthology.orchestra.run.smithi163.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout: fi 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:done 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:fi 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:fi 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T06:01:16.834 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T06:01:16.835 INFO:teuthology.orchestra.run.smithi163.stdout:fi 2024-07-26T06:01:16.836 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-07-26T06:01:16.912 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-07-26T06:01:16.912 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-07-26T06:01:16.912 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-07-26T06:01:16.912 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 68.el9 2024-07-26T06:01:16.912 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8264254 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-07-26T06:01:16.913 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T06:01:16.915 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T06:01:16.915 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 6.10.0-g3e9255e90710 2024-07-26T06:01:16.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T06:01:17.455 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-07-26T06:01:18.946 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T06:01:18.980 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-07-26T06:01:18.983 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-07-26T06:01:19.003 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.ftMAAV0bCX 2024-07-26T06:01:19.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ftMAAV0bCX 2024-07-26T06:01:19.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.ftMAAV0bCX 2024-07-26T06:01:19.209 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.ftMAAV0bCX is 6KB 2024-07-26T06:01:19.258 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.ftMAAV0bCX 2024-07-26T06:01:19.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T06:01:19.334 INFO:teuthology.orchestra.run.smithi163.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T06:01:19.335 INFO:teuthology.orchestra.run.smithi163.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T06:01:19.335 INFO:teuthology.orchestra.run.smithi163.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T06:01:19.335 INFO:teuthology.orchestra.run.smithi163.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T06:01:19.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T06:01:19.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-07-26T06:01:45.387 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-07-26T06:01:45.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:01:45.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:01:49.506 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-26T06:01:49.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:01:49.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:02:19.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-07-26T06:02:24.099 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-07-26T06:02:29.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:02:29.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:02:33.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:02:33.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:02:36.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-07-26T06:02:47.391 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-07-26T06:02:48.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:02:48.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:02:59.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:02:59.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:03:02.904 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-07-26T06:03:03.486 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:03:03.486 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.10.0-g3e9255e90710"... 2024-07-26T06:03:03.487 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-07-26T06:03:03.525 INFO:teuthology.orchestra.run.smithi164.stdout:6.10.0-g3e9255e90710 2024-07-26T06:03:03.525 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T06:03:03.525 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T06:03:03.525 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T06:03:04.526 DEBUG:teuthology.parallel:result is None 2024-07-26T06:03:06.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-07-26T06:03:21.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:03:21.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:03:24.371 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-07-26T06:03:25.003 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:03:25.003 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.10.0-g3e9255e90710"... 2024-07-26T06:03:25.003 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-07-26T06:03:25.040 INFO:teuthology.orchestra.run.smithi163.stdout:6.10.0-g3e9255e90710 2024-07-26T06:03:25.041 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T06:03:25.041 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T06:03:25.041 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T06:03:26.044 DEBUG:teuthology.parallel:result is None 2024-07-26T06:03:26.044 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T06:03:26.053 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T06:03:26.054 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T06:03:26.062 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T06:03:26.122 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T06:03:26.131 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T06:03:26.216 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T06:03:26.217 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T06:03:26.224 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T06:03:26.319 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T06:03:26.344 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T06:03:26.345 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T06:03:26.352 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T06:03:26.491 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:03:26.513 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:03:26.539 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:03:26.563 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:03:26.568 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T06:03:26.582 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T06:03:26.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T06:03:26.586 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T06:03:26.676 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T06:03:26.690 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T06:03:26.690 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T06:03:26.694 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T06:03:26.750 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:03:26.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:03:26.926 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:03:26.997 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:03:27.067 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:03:27.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T06:03:27.151 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:03:27.227 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:03:27.327 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:03:27.411 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:03:27.491 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:03:27.491 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T06:03:27.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-07-26T06:03:27.571 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-07-26T06:03:27.668 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:03:27.724 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:03:28.017 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T06:03:28.028 INFO:teuthology.task.internal:Starting timer... 2024-07-26T06:03:28.028 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T06:03:28.138 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T06:03:28.239 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-07-26T06:03:28.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-07-26T06:03:28.361 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-07-26T06:03:28.365 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-07-26T06:03:28.478 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-07-26T06:03:28.482 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T06:03:28.482 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-07-26T06:03:28.546 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-07-26T06:03:28.546 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-07-26T06:03:28.599 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-07-26T06:03:28.600 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-07-26T06:03:28.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:03:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:03:28.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:03:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:03:28.741 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T06:03:28.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-07-26T06:03:28.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-07-26T06:03:28.902 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T06:03:28.913 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T06:03:28.916 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T06:03:29.031 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T06:03:29.046 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T06:03:29.047 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi163.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T06:10:24.840 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-07-26T06:10:24.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:10:24.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:10:24.970 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-07-26T06:10:25.126 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:10:25.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:10:25.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:10:25.229 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-07-26T06:10:25.381 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:10:25.382 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T06:10:25.393 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T06:10:25.393 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T06:10:25.394 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:10:25.402 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T06:10:25.402 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:10:25.486 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T06:10:25.490 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T06:10:25.526 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T06:10:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T06:10:25.602 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-07-26T06:10:25.607 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-07-26T06:10:25.634 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-07-26T06:10:25.638 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-07-26T06:10:25.677 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T06:10:25.677 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T06:10:25.718 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T06:10:25.718 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T06:10:25.853 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:10:25.863 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:10:25.863 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-07-26T06:10:25.863 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.863 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.863 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.863 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.864 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:10:25.873 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:10:25.873 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-07-26T06:10:25.873 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.873 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.873 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.873 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:10:25.874 INFO:teuthology.run_tasks:Running task install... 2024-07-26T06:10:25.883 DEBUG:teuthology.task.install:project ceph 2024-07-26T06:10:25.883 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708'}} 2024-07-26T06:10:25.883 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708'} 2024-07-26T06:10:25.884 INFO:teuthology.task.install:Using flavor: default 2024-07-26T06:10:25.889 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T06:10:25.889 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T06:10:25.890 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708', 'tag': None, 'wait_for_package': False} 2024-07-26T06:10:25.890 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4aae82f07003420ccc152fe0b01f3d5a44885708 2024-07-26T06:10:25.893 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708', 'tag': None, 'wait_for_package': False} 2024-07-26T06:10:25.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4aae82f07003420ccc152fe0b01f3d5a44885708 2024-07-26T06:10:26.036 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/ 2024-07-26T06:10:26.036 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4588.g4aae82f0 2024-07-26T06:10:26.043 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/ 2024-07-26T06:10:26.043 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4588.g4aae82f0 2024-07-26T06:10:26.146 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T06:10:26.146 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:10:26.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T06:10:26.151 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-mgr-vol-clone-stats-bugfixes/4aae82f07003420ccc152fe0b01f3d5a44885708/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T06:10:26.151 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:10:26.151 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T06:10:26.235 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T06:10:26.235 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4aae82f07003420ccc152fe0b01f3d5a44885708/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T06:10:26.239 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T06:10:26.239 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4aae82f07003420ccc152fe0b01f3d5a44885708/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T06:10:26.364 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T06:10:26.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T06:10:26.475 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T06:10:26.478 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T06:10:26.553 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-07-26T06:10:26.554 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-07-26T06:10:26.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-07-26T06:10:26.559 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-07-26T06:10:26.881 INFO:teuthology.orchestra.run.smithi164.stdout:45 files removed 2024-07-26T06:10:26.883 INFO:teuthology.orchestra.run.smithi163.stdout:45 files removed 2024-07-26T06:10:26.919 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T06:10:26.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T06:10:27.703 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 235 kB/s | 84 kB 00:00 2024-07-26T06:10:27.714 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 223 kB/s | 84 kB 00:00 2024-07-26T06:10:28.179 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 63 kB/s | 24 kB 00:00 2024-07-26T06:10:28.182 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 64 kB/s | 24 kB 00:00 2024-07-26T06:10:28.612 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-07-26T06:10:28.640 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-07-26T06:10:30.629 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-07-26T06:10:31.852 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.2 MB 00:03 2024-07-26T06:10:33.165 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-07-26T06:10:35.553 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 20 MB 00:02 2024-07-26T06:10:37.898 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.5 MB 00:00 2024-07-26T06:10:39.524 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 18 kB 00:00 2024-07-26T06:10:40.763 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-07-26T06:10:42.344 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-07-26T06:10:44.632 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 18 kB 00:00 2024-07-26T06:10:45.894 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-07-26T06:10:47.175 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-07-26T06:10:49.751 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:10:49.753 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:10:49.773 INFO:teuthology.orchestra.run.smithi163.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T06:10:49.774 INFO:teuthology.orchestra.run.smithi163.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T06:10:49.775 INFO:teuthology.orchestra.run.smithi163.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T06:10:49.776 INFO:teuthology.orchestra.run.smithi163.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T06:10:49.779 INFO:teuthology.orchestra.run.smithi163.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T06:10:49.782 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T06:10:49.783 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T06:10:49.856 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 6.6 k 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 5.4 M 2024-07-26T06:10:49.881 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 893 k 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 145 k 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 1.5 M 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 143 k 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 14 M 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 7.4 M 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 49 k 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 11 M 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 49 M 2024-07-26T06:10:49.882 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 274 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 764 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 228 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 27 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 18 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 32 k 2024-07-26T06:10:49.883 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 749 k 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 127 k 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 187 k 2024-07-26T06:10:49.884 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 372 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 350 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 116 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 85 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 3.1 M 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 170 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T06:10:49.885 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 3.4 M 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 3.2 M 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T06:10:49.886 INFO:teuthology.orchestra.run.smithi163.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 21 M 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 34 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 2.4 M 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 280 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 4.7 M 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 17 M 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 16 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 25 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T06:10:49.887 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 163 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T06:10:49.888 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 504 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 5.4 M 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T06:10:49.889 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 45 k 2024-07-26T06:10:49.890 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 136 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T06:10:49.891 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T06:10:49.892 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T06:10:49.893 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T06:10:49.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T06:10:49.895 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T06:10:49.896 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout:Install 148 Packages 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 12 Packages 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:10:49.897 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 239 M 2024-07-26T06:10:49.898 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-07-26T06:10:50.902 INFO:teuthology.orchestra.run.smithi163.stdout:(1/160): ceph-19.0.0-4588.g4aae82f0.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-07-26T06:10:51.102 INFO:teuthology.orchestra.run.smithi163.stdout:(2/160): ceph-fuse-19.0.0-4588.g4aae82f0.el9.x8 4.4 MB/s | 893 kB 00:00 2024-07-26T06:10:51.169 INFO:teuthology.orchestra.run.smithi163.stdout:(3/160): ceph-immutable-object-cache-19.0.0-458 2.2 MB/s | 145 kB 00:00 2024-07-26T06:10:51.327 INFO:teuthology.orchestra.run.smithi163.stdout:(4/160): ceph-base-19.0.0-4588.g4aae82f0.el9.x8 7.9 MB/s | 5.4 MB 00:00 2024-07-26T06:10:51.420 INFO:teuthology.orchestra.run.smithi163.stdout:(5/160): ceph-mds-19.0.0-4588.g4aae82f0.el9.x86 9.5 MB/s | 2.4 MB 00:00 2024-07-26T06:10:51.495 INFO:teuthology.orchestra.run.smithi163.stdout:(6/160): ceph-mgr-19.0.0-4588.g4aae82f0.el9.x86 9.1 MB/s | 1.5 MB 00:00 2024-07-26T06:10:51.770 INFO:teuthology.orchestra.run.smithi163.stdout:(7/160): ceph-mon-19.0.0-4588.g4aae82f0.el9.x86 13 MB/s | 4.7 MB 00:00 2024-07-26T06:10:52.205 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-07-26T06:10:52.287 INFO:teuthology.orchestra.run.smithi163.stdout:(8/160): ceph-common-19.0.0-4588.g4aae82f0.el9. 13 MB/s | 21 MB 00:01 2024-07-26T06:10:52.363 INFO:teuthology.orchestra.run.smithi163.stdout:(9/160): ceph-selinux-19.0.0-4588.g4aae82f0.el9 340 kB/s | 25 kB 00:00 2024-07-26T06:10:52.638 INFO:teuthology.orchestra.run.smithi163.stdout:(10/160): ceph-osd-19.0.0-4588.g4aae82f0.el9.x8 15 MB/s | 17 MB 00:01 2024-07-26T06:10:52.721 INFO:teuthology.orchestra.run.smithi163.stdout:(11/160): cephfs-mirror-19.0.0-4588.g4aae82f0.e 2.7 MB/s | 228 kB 00:00 2024-07-26T06:10:52.913 INFO:teuthology.orchestra.run.smithi163.stdout:(12/160): ceph-radosgw-19.0.0-4588.g4aae82f0.el 9.4 MB/s | 11 MB 00:01 2024-07-26T06:10:52.939 INFO:teuthology.orchestra.run.smithi163.stdout:(13/160): cephfs-shell-19.0.0-4588.g4aae82f0.el 125 kB/s | 27 kB 00:00 2024-07-26T06:10:52.964 INFO:teuthology.orchestra.run.smithi163.stdout:(14/160): libcephfs-devel-19.0.0-4588.g4aae82f0 646 kB/s | 32 kB 00:00 2024-07-26T06:10:53.022 INFO:teuthology.orchestra.run.smithi163.stdout:(15/160): libcephfs2-19.0.0-4588.g4aae82f0.el9. 8.9 MB/s | 749 kB 00:00 2024-07-26T06:10:53.056 INFO:teuthology.orchestra.run.smithi163.stdout:(16/160): libcephsqlite-19.0.0-4588.g4aae82f0.e 1.8 MB/s | 163 kB 00:00 2024-07-26T06:10:53.090 INFO:teuthology.orchestra.run.smithi163.stdout:(17/160): librados-devel-19.0.0-4588.g4aae82f0. 1.9 MB/s | 127 kB 00:00 2024-07-26T06:10:53.148 INFO:teuthology.orchestra.run.smithi163.stdout:(18/160): libradosstriper1-19.0.0-4588.g4aae82f 5.4 MB/s | 504 kB 00:00 2024-07-26T06:10:53.199 INFO:teuthology.orchestra.run.smithi163.stdout:(19/160): python3-ceph-argparse-19.0.0-4588.g4a 920 kB/s | 45 kB 00:00 2024-07-26T06:10:53.276 INFO:teuthology.orchestra.run.smithi163.stdout:(20/160): python3-ceph-common-19.0.0-4588.g4aae 1.7 MB/s | 136 kB 00:00 2024-07-26T06:10:53.341 INFO:teuthology.orchestra.run.smithi163.stdout:(21/160): python3-cephfs-19.0.0-4588.g4aae82f0. 2.9 MB/s | 187 kB 00:00 2024-07-26T06:10:53.466 INFO:teuthology.orchestra.run.smithi163.stdout:(22/160): librgw2-19.0.0-4588.g4aae82f0.el9.x86 14 MB/s | 5.4 MB 00:00 2024-07-26T06:10:53.508 INFO:teuthology.orchestra.run.smithi163.stdout:(23/160): python3-rados-19.0.0-4588.g4aae82f0.e 2.2 MB/s | 372 kB 00:00 2024-07-26T06:10:53.550 INFO:teuthology.orchestra.run.smithi163.stdout:(24/160): python3-rbd-19.0.0-4588.g4aae82f0.el9 4.1 MB/s | 350 kB 00:00 2024-07-26T06:10:53.584 INFO:teuthology.orchestra.run.smithi163.stdout:(25/160): python3-rgw-19.0.0-4588.g4aae82f0.el9 1.5 MB/s | 116 kB 00:00 2024-07-26T06:10:53.609 INFO:teuthology.orchestra.run.smithi163.stdout:(26/160): rbd-fuse-19.0.0-4588.g4aae82f0.el9.x8 1.4 MB/s | 85 kB 00:00 2024-07-26T06:10:53.668 INFO:teuthology.orchestra.run.smithi163.stdout:(27/160): rbd-nbd-19.0.0-4588.g4aae82f0.el9.x86 2.9 MB/s | 170 kB 00:00 2024-07-26T06:10:53.718 INFO:teuthology.orchestra.run.smithi163.stdout:(28/160): ceph-grafana-dashboards-19.0.0-4588.g 694 kB/s | 34 kB 00:00 2024-07-26T06:10:53.793 INFO:teuthology.orchestra.run.smithi163.stdout:(29/160): rbd-mirror-19.0.0-4588.g4aae82f0.el9. 15 MB/s | 3.1 MB 00:00 2024-07-26T06:10:53.845 INFO:teuthology.orchestra.run.smithi163.stdout:(30/160): ceph-mgr-cephadm-19.0.0-4588.g4aae82f 1.1 MB/s | 143 kB 00:00 2024-07-26T06:10:54.294 INFO:teuthology.orchestra.run.smithi163.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-07-26T06:10:54.361 INFO:teuthology.orchestra.run.smithi163.stdout:(32/160): ceph-mgr-modules-core-19.0.0-4588.g4a 4.2 MB/s | 280 kB 00:00 2024-07-26T06:10:54.436 INFO:teuthology.orchestra.run.smithi163.stdout:(33/160): ceph-mgr-rook-19.0.0-4588.g4aae82f0.e 664 kB/s | 49 kB 00:00 2024-07-26T06:10:54.637 INFO:teuthology.orchestra.run.smithi163.stdout:(34/160): ceph-mgr-dashboard-19.0.0-4588.g4aae8 17 MB/s | 14 MB 00:00 2024-07-26T06:10:54.662 INFO:teuthology.orchestra.run.smithi163.stdout:(35/160): ceph-prometheus-alerts-19.0.0-4588.g4 73 kB/s | 16 kB 00:00 2024-07-26T06:10:54.696 INFO:teuthology.orchestra.run.smithi163.stdout:(36/160): ceph-volume-19.0.0-4588.g4aae82f0.el9 4.7 MB/s | 274 kB 00:00 2024-07-26T06:10:54.747 INFO:teuthology.orchestra.run.smithi163.stdout:(37/160): cephadm-19.0.0-4588.g4aae82f0.el9.noa 9.0 MB/s | 764 kB 00:00 2024-07-26T06:10:54.764 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:10:54.766 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:10:54.771 INFO:teuthology.orchestra.run.smithi163.stdout:(38/160): cephfs-top-19.0.0-4588.g4aae82f0.el9. 245 kB/s | 18 kB 00:00 2024-07-26T06:10:54.786 INFO:teuthology.orchestra.run.smithi164.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T06:10:54.787 INFO:teuthology.orchestra.run.smithi164.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T06:10:54.788 INFO:teuthology.orchestra.run.smithi164.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T06:10:54.789 INFO:teuthology.orchestra.run.smithi164.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T06:10:54.792 INFO:teuthology.orchestra.run.smithi164.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T06:10:54.795 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T06:10:54.796 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T06:10:54.870 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-07-26T06:10:54.890 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-07-26T06:10:54.890 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-07-26T06:10:54.890 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-07-26T06:10:54.890 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 6.6 k 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 5.4 M 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 893 k 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 145 k 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 1.5 M 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 143 k 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 14 M 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 7.4 M 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 49 k 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 11 M 2024-07-26T06:10:54.891 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 49 M 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 274 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 764 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 228 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 27 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 18 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 32 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 749 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 127 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T06:10:54.892 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 187 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 372 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 350 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 116 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 85 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 3.1 M 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 170 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T06:10:54.893 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 3.4 M 2024-07-26T06:10:54.894 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 3.2 M 2024-07-26T06:10:54.895 INFO:teuthology.orchestra.run.smithi164.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T06:10:54.895 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T06:10:54.895 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T06:10:54.895 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-07-26T06:10:54.895 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 21 M 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 34 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 2.4 M 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 280 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 4.7 M 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 17 M 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4588.g4aae82f0.el9 ceph-noarch 16 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 25 k 2024-07-26T06:10:54.896 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 163 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T06:10:54.897 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 504 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 5.4 M 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T06:10:54.898 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 45 k 2024-07-26T06:10:54.899 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-4588.g4aae82f0.el9 ceph 136 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T06:10:54.900 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T06:10:54.901 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T06:10:54.902 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T06:10:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T06:10:54.904 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout:Install 148 Packages 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 12 Packages 2024-07-26T06:10:54.905 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:10:54.906 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 239 M 2024-07-26T06:10:54.906 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-07-26T06:10:54.946 INFO:teuthology.orchestra.run.smithi163.stdout:(39/160): fuse-2.9.9-16.el9.x86_64.rpm 401 kB/s | 80 kB 00:00 2024-07-26T06:10:55.064 INFO:teuthology.orchestra.run.smithi163.stdout:(40/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-07-26T06:10:55.106 INFO:teuthology.orchestra.run.smithi163.stdout:(41/160): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-07-26T06:10:55.156 INFO:teuthology.orchestra.run.smithi163.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-07-26T06:10:55.214 INFO:teuthology.orchestra.run.smithi163.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-07-26T06:10:55.275 INFO:teuthology.orchestra.run.smithi163.stdout:(44/160): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-07-26T06:10:55.332 INFO:teuthology.orchestra.run.smithi163.stdout:(45/160): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-07-26T06:10:55.390 INFO:teuthology.orchestra.run.smithi163.stdout:(46/160): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-07-26T06:10:55.474 INFO:teuthology.orchestra.run.smithi163.stdout:(47/160): python3-cryptography-36.0.1-4.el9.x86 3.3 MB/s | 1.2 MB 00:00 2024-07-26T06:10:55.631 INFO:teuthology.orchestra.run.smithi164.stdout:(1/160): ceph-19.0.0-4588.g4aae82f0.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-07-26T06:10:55.632 INFO:teuthology.orchestra.run.smithi163.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 689 kB 00:00 2024-07-26T06:10:55.786 INFO:teuthology.orchestra.run.smithi163.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 701 kB/s | 107 kB 00:00 2024-07-26T06:10:55.823 INFO:teuthology.orchestra.run.smithi164.stdout:(2/160): ceph-fuse-19.0.0-4588.g4aae82f0.el9.x8 4.6 MB/s | 893 kB 00:00 2024-07-26T06:10:55.833 INFO:teuthology.orchestra.run.smithi163.stdout:(50/160): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-07-26T06:10:55.859 INFO:teuthology.orchestra.run.smithi163.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 306 kB/s | 22 kB 00:00 2024-07-26T06:10:55.909 INFO:teuthology.orchestra.run.smithi163.stdout:(52/160): flexiblas-3.0.4-8.el9.x86_64.rpm 664 kB/s | 33 kB 00:00 2024-07-26T06:10:55.915 INFO:teuthology.orchestra.run.smithi164.stdout:(3/160): ceph-immutable-object-cache-19.0.0-458 1.6 MB/s | 145 kB 00:00 2024-07-26T06:10:55.942 INFO:teuthology.orchestra.run.smithi163.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-07-26T06:10:56.051 INFO:teuthology.orchestra.run.smithi163.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 169 kB/s | 18 kB 00:00 2024-07-26T06:10:56.057 INFO:teuthology.orchestra.run.smithi164.stdout:(4/160): ceph-base-19.0.0-4588.g4aae82f0.el9.x8 7.9 MB/s | 5.4 MB 00:00 2024-07-26T06:10:56.110 INFO:teuthology.orchestra.run.smithi163.stdout:(55/160): libaio-devel-0.3.111-13.el9.x86_64.rp 194 kB/s | 11 kB 00:00 2024-07-26T06:10:56.174 INFO:teuthology.orchestra.run.smithi164.stdout:(5/160): ceph-mds-19.0.0-4588.g4aae82f0.el9.x86 9.2 MB/s | 2.4 MB 00:00 2024-07-26T06:10:56.224 INFO:teuthology.orchestra.run.smithi164.stdout:(6/160): ceph-mgr-19.0.0-4588.g4aae82f0.el9.x86 9.1 MB/s | 1.5 MB 00:00 2024-07-26T06:10:56.226 INFO:teuthology.orchestra.run.smithi163.stdout:(56/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-07-26T06:10:56.285 INFO:teuthology.orchestra.run.smithi163.stdout:(57/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 789 kB/s | 45 kB 00:00 2024-07-26T06:10:56.444 INFO:teuthology.orchestra.run.smithi163.stdout:(58/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.6 MB/s | 3.0 MB 00:00 2024-07-26T06:10:56.478 INFO:teuthology.orchestra.run.smithi163.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-07-26T06:10:56.528 INFO:teuthology.orchestra.run.smithi163.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 519 kB/s | 25 kB 00:00 2024-07-26T06:10:56.570 INFO:teuthology.orchestra.run.smithi163.stdout:(61/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-07-26T06:10:56.603 INFO:teuthology.orchestra.run.smithi163.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 3.6 MB/s | 581 kB 00:00 2024-07-26T06:10:56.662 INFO:teuthology.orchestra.run.smithi163.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-07-26T06:10:56.696 INFO:teuthology.orchestra.run.smithi163.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-07-26T06:10:56.702 INFO:teuthology.orchestra.run.smithi164.stdout:(7/160): ceph-mon-19.0.0-4588.g4aae82f0.el9.x86 8.9 MB/s | 4.7 MB 00:00 2024-07-26T06:10:56.754 INFO:teuthology.orchestra.run.smithi163.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 387 kB/s | 35 kB 00:00 2024-07-26T06:10:57.075 INFO:teuthology.orchestra.run.smithi164.stdout:(8/160): ceph-common-19.0.0-4588.g4aae82f0.el9. 12 MB/s | 21 MB 00:01 2024-07-26T06:10:57.126 INFO:teuthology.orchestra.run.smithi164.stdout:(9/160): ceph-selinux-19.0.0-4588.g4aae82f0.el9 499 kB/s | 25 kB 00:00 2024-07-26T06:10:57.392 INFO:teuthology.orchestra.run.smithi164.stdout:(10/160): ceph-osd-19.0.0-4588.g4aae82f0.el9.x8 14 MB/s | 17 MB 00:01 2024-07-26T06:10:57.450 INFO:teuthology.orchestra.run.smithi164.stdout:(11/160): cephfs-mirror-19.0.0-4588.g4aae82f0.e 3.9 MB/s | 228 kB 00:00 2024-07-26T06:10:57.463 INFO:teuthology.orchestra.run.smithi163.stdout:(66/160): ceph-test-19.0.0-4588.g4aae82f0.el9.x 9.7 MB/s | 49 MB 00:05 2024-07-26T06:10:57.540 INFO:teuthology.orchestra.run.smithi164.stdout:(12/160): cephfs-shell-19.0.0-4588.g4aae82f0.el 306 kB/s | 27 kB 00:00 2024-07-26T06:10:57.593 INFO:teuthology.orchestra.run.smithi164.stdout:(13/160): libcephfs-devel-19.0.0-4588.g4aae82f0 610 kB/s | 32 kB 00:00 2024-07-26T06:10:57.677 INFO:teuthology.orchestra.run.smithi164.stdout:(14/160): libcephfs2-19.0.0-4588.g4aae82f0.el9. 8.9 MB/s | 749 kB 00:00 2024-07-26T06:10:57.705 INFO:teuthology.orchestra.run.smithi163.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 4.7 MB/s | 4.7 MB 00:01 2024-07-26T06:10:57.852 INFO:teuthology.orchestra.run.smithi164.stdout:(15/160): ceph-radosgw-19.0.0-4588.g4aae82f0.el 9.3 MB/s | 11 MB 00:01 2024-07-26T06:10:57.864 INFO:teuthology.orchestra.run.smithi163.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-07-26T06:10:57.885 INFO:teuthology.orchestra.run.smithi164.stdout:(16/160): libcephsqlite-19.0.0-4588.g4aae82f0.e 786 kB/s | 163 kB 00:00 2024-07-26T06:10:57.914 INFO:teuthology.orchestra.run.smithi163.stdout:(69/160): postgresql-13.14-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-07-26T06:10:57.919 INFO:teuthology.orchestra.run.smithi164.stdout:(17/160): librados-devel-19.0.0-4588.g4aae82f0. 1.9 MB/s | 127 kB 00:00 2024-07-26T06:10:57.947 INFO:teuthology.orchestra.run.smithi163.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-07-26T06:10:57.969 INFO:teuthology.orchestra.run.smithi164.stdout:(18/160): libradosstriper1-19.0.0-4588.g4aae82f 6.0 MB/s | 504 kB 00:00 2024-07-26T06:10:58.006 INFO:teuthology.orchestra.run.smithi163.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 3.0 MB/s | 911 kB 00:00 2024-07-26T06:10:58.020 INFO:teuthology.orchestra.run.smithi164.stdout:(19/160): python3-ceph-argparse-19.0.0-4588.g4a 921 kB/s | 45 kB 00:00 2024-07-26T06:10:58.064 INFO:teuthology.orchestra.run.smithi163.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-07-26T06:10:58.078 INFO:teuthology.orchestra.run.smithi164.stdout:(20/160): python3-ceph-common-19.0.0-4588.g4aae 2.3 MB/s | 136 kB 00:00 2024-07-26T06:10:58.137 INFO:teuthology.orchestra.run.smithi164.stdout:(21/160): python3-cephfs-19.0.0-4588.g4aae82f0. 3.2 MB/s | 187 kB 00:00 2024-07-26T06:10:58.148 INFO:teuthology.orchestra.run.smithi163.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-07-26T06:10:58.204 INFO:teuthology.orchestra.run.smithi164.stdout:(22/160): python3-rados-19.0.0-4588.g4aae82f0.e 5.5 MB/s | 372 kB 00:00 2024-07-26T06:10:58.207 INFO:teuthology.orchestra.run.smithi163.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-07-26T06:10:58.291 INFO:teuthology.orchestra.run.smithi163.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-07-26T06:10:58.329 INFO:teuthology.orchestra.run.smithi164.stdout:(23/160): librgw2-19.0.0-4588.g4aae82f0.el9.x86 13 MB/s | 5.4 MB 00:00 2024-07-26T06:10:58.396 INFO:teuthology.orchestra.run.smithi164.stdout:(24/160): python3-rbd-19.0.0-4588.g4aae82f0.el9 1.8 MB/s | 350 kB 00:00 2024-07-26T06:10:58.430 INFO:teuthology.orchestra.run.smithi164.stdout:(25/160): python3-rgw-19.0.0-4588.g4aae82f0.el9 1.1 MB/s | 116 kB 00:00 2024-07-26T06:10:58.455 INFO:teuthology.orchestra.run.smithi164.stdout:(26/160): rbd-fuse-19.0.0-4588.g4aae82f0.el9.x8 1.4 MB/s | 85 kB 00:00 2024-07-26T06:10:58.508 INFO:teuthology.orchestra.run.smithi163.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-07-26T06:10:58.513 INFO:teuthology.orchestra.run.smithi164.stdout:(27/160): rbd-nbd-19.0.0-4588.g4aae82f0.el9.x86 2.9 MB/s | 170 kB 00:00 2024-07-26T06:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout:(28/160): ceph-grafana-dashboards-19.0.0-4588.g 841 kB/s | 34 kB 00:00 2024-07-26T06:10:58.585 INFO:teuthology.orchestra.run.smithi163.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 336 kB/s | 26 kB 00:00 2024-07-26T06:10:58.606 INFO:teuthology.orchestra.run.smithi164.stdout:(29/160): ceph-mgr-cephadm-19.0.0-4588.g4aae82f 2.8 MB/s | 143 kB 00:00 2024-07-26T06:10:58.658 INFO:teuthology.orchestra.run.smithi163.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 267 kB/s | 19 kB 00:00 2024-07-26T06:10:58.689 INFO:teuthology.orchestra.run.smithi164.stdout:(30/160): rbd-mirror-19.0.0-4588.g4aae82f0.el9. 12 MB/s | 3.1 MB 00:00 2024-07-26T06:10:58.825 INFO:teuthology.orchestra.run.smithi163.stdout:(79/160): python3-babel-2.9.1-2.el9.noarch.rpm 7.9 MB/s | 6.0 MB 00:00 2024-07-26T06:10:58.850 INFO:teuthology.orchestra.run.smithi163.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 901 kB/s | 172 kB 00:00 2024-07-26T06:10:58.884 INFO:teuthology.orchestra.run.smithi163.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-07-26T06:10:59.018 INFO:teuthology.orchestra.run.smithi163.stdout:(82/160): postgresql-server-13.14-1.el9.x86_64. 5.2 MB/s | 5.8 MB 00:01 2024-07-26T06:10:59.051 INFO:teuthology.orchestra.run.smithi163.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-07-26T06:10:59.085 INFO:teuthology.orchestra.run.smithi163.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-07-26T06:10:59.135 INFO:teuthology.orchestra.run.smithi163.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 932 kB/s | 77 kB 00:00 2024-07-26T06:10:59.186 INFO:teuthology.orchestra.run.smithi163.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 423 kB/s | 42 kB 00:00 2024-07-26T06:10:59.219 INFO:teuthology.orchestra.run.smithi163.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-07-26T06:10:59.253 INFO:teuthology.orchestra.run.smithi163.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-07-26T06:10:59.287 INFO:teuthology.orchestra.run.smithi163.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-07-26T06:10:59.312 INFO:teuthology.orchestra.run.smithi163.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 882 kB/s | 51 kB 00:00 2024-07-26T06:10:59.346 INFO:teuthology.orchestra.run.smithi163.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 931 kB/s | 54 kB 00:00 2024-07-26T06:10:59.356 INFO:teuthology.orchestra.run.smithi164.stdout:(31/160): ceph-mgr-dashboard-19.0.0-4588.g4aae8 19 MB/s | 14 MB 00:00 2024-07-26T06:10:59.404 INFO:teuthology.orchestra.run.smithi163.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 726 kB/s | 42 kB 00:00 2024-07-26T06:10:59.463 INFO:teuthology.orchestra.run.smithi163.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 769 kB/s | 44 kB 00:00 2024-07-26T06:10:59.481 INFO:teuthology.orchestra.run.smithi164.stdout:(32/160): ceph-mgr-diskprediction-local-19.0.0- 9.4 MB/s | 7.4 MB 00:00 2024-07-26T06:10:59.515 INFO:teuthology.orchestra.run.smithi164.stdout:(33/160): ceph-mgr-modules-core-19.0.0-4588.g4a 1.7 MB/s | 280 kB 00:00 2024-07-26T06:10:59.521 INFO:teuthology.orchestra.run.smithi163.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 3.8 MB/s | 221 kB 00:00 2024-07-26T06:10:59.542 INFO:teuthology.orchestra.run.smithi164.stdout:(34/160): ceph-mgr-rook-19.0.0-4588.g4aae82f0.e 852 kB/s | 49 kB 00:00 2024-07-26T06:10:59.566 INFO:teuthology.orchestra.run.smithi164.stdout:(35/160): ceph-prometheus-alerts-19.0.0-4588.g4 332 kB/s | 16 kB 00:00 2024-07-26T06:10:59.580 INFO:teuthology.orchestra.run.smithi163.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 625 kB/s | 36 kB 00:00 2024-07-26T06:10:59.601 INFO:teuthology.orchestra.run.smithi164.stdout:(36/160): ceph-volume-19.0.0-4588.g4aae82f0.el9 4.7 MB/s | 274 kB 00:00 2024-07-26T06:10:59.630 INFO:teuthology.orchestra.run.smithi163.stdout:(96/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 964 kB/s | 47 kB 00:00 2024-07-26T06:10:59.666 INFO:teuthology.orchestra.run.smithi164.stdout:(37/160): cephadm-19.0.0-4588.g4aae82f0.el9.noa 7.5 MB/s | 764 kB 00:00 2024-07-26T06:10:59.692 INFO:teuthology.orchestra.run.smithi164.stdout:(38/160): cephfs-top-19.0.0-4588.g4aae82f0.el9. 200 kB/s | 18 kB 00:00 2024-07-26T06:10:59.697 INFO:teuthology.orchestra.run.smithi163.stdout:(97/160): socat-1.7.4.1-6.el9.x86_64.rpm 4.5 MB/s | 304 kB 00:00 2024-07-26T06:10:59.797 INFO:teuthology.orchestra.run.smithi163.stdout:(98/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.4 MB/s | 5.1 MB 00:00 2024-07-26T06:10:59.800 INFO:teuthology.orchestra.run.smithi164.stdout:(39/160): fuse-2.9.9-16.el9.x86_64.rpm 599 kB/s | 80 kB 00:00 2024-07-26T06:10:59.815 INFO:teuthology.orchestra.run.smithi163.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 497 kB/s | 58 kB 00:00 2024-07-26T06:10:59.857 INFO:teuthology.orchestra.run.smithi163.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-07-26T06:10:59.874 INFO:teuthology.orchestra.run.smithi163.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-07-26T06:10:59.907 INFO:teuthology.orchestra.run.smithi163.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 910 kB/s | 45 kB 00:00 2024-07-26T06:10:59.958 INFO:teuthology.orchestra.run.smithi163.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 270 kB/s | 22 kB 00:00 2024-07-26T06:10:59.959 INFO:teuthology.orchestra.run.smithi164.stdout:(40/160): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-07-26T06:11:00.083 INFO:teuthology.orchestra.run.smithi163.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.3 MB/s | 548 kB 00:00 2024-07-26T06:11:00.133 INFO:teuthology.orchestra.run.smithi163.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 6.1 MB/s | 309 kB 00:00 2024-07-26T06:11:00.158 INFO:teuthology.orchestra.run.smithi163.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 803 kB/s | 19 kB 00:00 2024-07-26T06:11:00.192 INFO:teuthology.orchestra.run.smithi163.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.0 MB/s | 863 kB 00:00 2024-07-26T06:11:00.326 INFO:teuthology.orchestra.run.smithi163.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 196 kB/s | 26 kB 00:00 2024-07-26T06:11:00.351 INFO:teuthology.orchestra.run.smithi163.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-07-26T06:11:00.385 INFO:teuthology.orchestra.run.smithi163.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-07-26T06:11:00.418 INFO:teuthology.orchestra.run.smithi163.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.6 MB/s | 151 kB 00:00 2024-07-26T06:11:00.493 INFO:teuthology.orchestra.run.smithi163.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-07-26T06:11:00.535 INFO:teuthology.orchestra.run.smithi164.stdout:(41/160): ceph-test-19.0.0-4588.g4aae82f0.el9.x 14 MB/s | 49 MB 00:03 2024-07-26T06:11:00.544 INFO:teuthology.orchestra.run.smithi163.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-07-26T06:11:00.560 INFO:teuthology.orchestra.run.smithi164.stdout:(42/160): libquadmath-11.4.1-3.el9.x86_64.rpm 258 kB/s | 195 kB 00:00 2024-07-26T06:11:00.578 INFO:teuthology.orchestra.run.smithi163.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-07-26T06:11:00.593 INFO:teuthology.orchestra.run.smithi164.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 400 kB/s | 253 kB 00:00 2024-07-26T06:11:00.603 INFO:teuthology.orchestra.run.smithi163.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T06:11:00.628 INFO:teuthology.orchestra.run.smithi163.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 586 kB/s | 14 kB 00:00 2024-07-26T06:11:00.669 INFO:teuthology.orchestra.run.smithi164.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 989 kB/s | 106 kB 00:00 2024-07-26T06:11:00.679 INFO:teuthology.orchestra.run.smithi163.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-07-26T06:11:00.693 INFO:teuthology.orchestra.run.smithi164.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-07-26T06:11:00.737 INFO:teuthology.orchestra.run.smithi163.stdout:(118/160): python3-cherrypy-18.6.1-2.el9.noarch 6.1 MB/s | 358 kB 00:00 2024-07-26T06:11:00.779 INFO:teuthology.orchestra.run.smithi163.stdout:(119/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.3 MB/s | 303 kB 00:00 2024-07-26T06:11:00.802 INFO:teuthology.orchestra.run.smithi164.stdout:(46/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-07-26T06:11:00.828 INFO:teuthology.orchestra.run.smithi164.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 791 kB/s | 125 kB 00:00 2024-07-26T06:11:00.846 INFO:teuthology.orchestra.run.smithi163.stdout:(120/160): python3-google-auth-2.32.0-1.el9.noa 3.2 MB/s | 216 kB 00:00 2024-07-26T06:11:00.928 INFO:teuthology.orchestra.run.smithi164.stdout:(48/160): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-07-26T06:11:00.972 INFO:teuthology.orchestra.run.smithi163.stdout:(121/160): libarrow-9.0.0-11.el9.x86_64.rpm 5.4 MB/s | 4.4 MB 00:00 2024-07-26T06:11:01.005 INFO:teuthology.orchestra.run.smithi163.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.4 MB/s | 144 kB 00:00 2024-07-26T06:11:01.030 INFO:teuthology.orchestra.run.smithi163.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 444 kB/s | 11 kB 00:00 2024-07-26T06:11:01.036 INFO:teuthology.orchestra.run.smithi164.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 689 kB 00:00 2024-07-26T06:11:01.056 INFO:teuthology.orchestra.run.smithi163.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 732 kB/s | 18 kB 00:00 2024-07-26T06:11:01.062 INFO:teuthology.orchestra.run.smithi164.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 802 kB/s | 107 kB 00:00 2024-07-26T06:11:01.081 INFO:teuthology.orchestra.run.smithi163.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 962 kB/s | 23 kB 00:00 2024-07-26T06:11:01.087 INFO:teuthology.orchestra.run.smithi164.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 439 kB/s | 22 kB 00:00 2024-07-26T06:11:01.106 INFO:teuthology.orchestra.run.smithi163.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 806 kB/s | 19 kB 00:00 2024-07-26T06:11:01.129 INFO:teuthology.orchestra.run.smithi164.stdout:(52/160): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-07-26T06:11:01.132 INFO:teuthology.orchestra.run.smithi163.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 816 kB/s | 20 kB 00:00 2024-07-26T06:11:01.154 INFO:teuthology.orchestra.run.smithi164.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 495 kB/s | 33 kB 00:00 2024-07-26T06:11:01.190 INFO:teuthology.orchestra.run.smithi164.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-07-26T06:11:01.213 INFO:teuthology.orchestra.run.smithi164.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 315 kB/s | 18 kB 00:00 2024-07-26T06:11:01.215 INFO:teuthology.orchestra.run.smithi163.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 5.5 MB/s | 2.0 MB 00:00 2024-07-26T06:11:01.238 INFO:teuthology.orchestra.run.smithi164.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 225 kB/s | 11 kB 00:00 2024-07-26T06:11:01.249 INFO:teuthology.orchestra.run.smithi163.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-07-26T06:11:01.272 INFO:teuthology.orchestra.run.smithi164.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.8 MB/s | 160 kB 00:00 2024-07-26T06:11:01.283 INFO:teuthology.orchestra.run.smithi163.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T06:11:01.297 INFO:teuthology.orchestra.run.smithi164.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 784 kB/s | 45 kB 00:00 2024-07-26T06:11:01.316 INFO:teuthology.orchestra.run.smithi163.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-07-26T06:11:01.348 INFO:teuthology.orchestra.run.smithi164.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-07-26T06:11:01.366 INFO:teuthology.orchestra.run.smithi163.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 4.4 MB/s | 1.0 MB 00:00 2024-07-26T06:11:01.389 INFO:teuthology.orchestra.run.smithi164.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-07-26T06:11:01.392 INFO:teuthology.orchestra.run.smithi163.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 681 kB/s | 16 kB 00:00 2024-07-26T06:11:01.415 INFO:teuthology.orchestra.run.smithi164.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 385 kB/s | 25 kB 00:00 2024-07-26T06:11:01.425 INFO:teuthology.orchestra.run.smithi163.stdout:(134/160): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-07-26T06:11:01.432 INFO:teuthology.orchestra.run.smithi164.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-07-26T06:11:01.465 INFO:teuthology.orchestra.run.smithi164.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.8 MB/s | 292 kB 00:00 2024-07-26T06:11:01.473 INFO:teuthology.orchestra.run.smithi163.stdout:(135/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-07-26T06:11:01.499 INFO:teuthology.orchestra.run.smithi164.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-07-26T06:11:01.509 INFO:teuthology.orchestra.run.smithi163.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 341 kB/s | 28 kB 00:00 2024-07-26T06:11:01.524 INFO:teuthology.orchestra.run.smithi164.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 611 kB/s | 35 kB 00:00 2024-07-26T06:11:01.535 INFO:teuthology.orchestra.run.smithi163.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 511 kB/s | 31 kB 00:00 2024-07-26T06:11:01.568 INFO:teuthology.orchestra.run.smithi163.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-07-26T06:11:01.594 INFO:teuthology.orchestra.run.smithi163.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-07-26T06:11:01.616 INFO:teuthology.orchestra.run.smithi164.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.1 MB/s | 3.0 MB 00:00 2024-07-26T06:11:01.629 INFO:teuthology.orchestra.run.smithi163.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-07-26T06:11:01.654 INFO:teuthology.orchestra.run.smithi163.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 852 kB/s | 51 kB 00:00 2024-07-26T06:11:01.688 INFO:teuthology.orchestra.run.smithi163.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 2.7 MB/s | 90 kB 00:00 2024-07-26T06:11:01.722 INFO:teuthology.orchestra.run.smithi163.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-07-26T06:11:01.725 INFO:teuthology.orchestra.run.smithi164.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-07-26T06:11:01.747 INFO:teuthology.orchestra.run.smithi163.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 828 kB/s | 20 kB 00:00 2024-07-26T06:11:01.789 INFO:teuthology.orchestra.run.smithi163.stdout:(145/160): re2-20211101-3.el9.x86_64.rpm 4.6 MB/s | 192 kB 00:00 2024-07-26T06:11:01.823 INFO:teuthology.orchestra.run.smithi163.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-07-26T06:11:01.842 INFO:teuthology.orchestra.run.smithi164.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-07-26T06:11:01.884 INFO:teuthology.orchestra.run.smithi164.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 5.6 MB/s | 911 kB 00:00 2024-07-26T06:11:01.984 INFO:teuthology.orchestra.run.smithi164.stdout:(70/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 9.7 MB/s | 4.4 MB 00:00 2024-07-26T06:11:02.001 INFO:teuthology.orchestra.run.smithi164.stdout:(71/160): postgresql-private-libs-13.14-1.el9.x 840 kB/s | 133 kB 00:00 2024-07-26T06:11:02.051 INFO:teuthology.orchestra.run.smithi164.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-07-26T06:11:02.065 INFO:teuthology.orchestra.run.smithi163.stdout:(147/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-07-26T06:11:02.101 INFO:teuthology.orchestra.run.smithi164.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-07-26T06:11:02.144 INFO:teuthology.orchestra.run.smithi164.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-07-26T06:11:02.182 INFO:teuthology.orchestra.run.smithi163.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-07-26T06:11:02.194 INFO:teuthology.orchestra.run.smithi164.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-07-26T06:11:02.257 INFO:teuthology.orchestra.run.smithi163.stdout:(149/160): librados2-19.0.0-4588.g4aae82f0.el9. 7.9 MB/s | 3.4 MB 00:00 2024-07-26T06:11:02.262 INFO:teuthology.orchestra.run.smithi164.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-07-26T06:11:02.311 INFO:teuthology.orchestra.run.smithi164.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 534 kB/s | 26 kB 00:00 2024-07-26T06:11:02.353 INFO:teuthology.orchestra.run.smithi164.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 468 kB/s | 19 kB 00:00 2024-07-26T06:11:02.433 INFO:teuthology.orchestra.run.smithi163.stdout:(150/160): librbd1-19.0.0-4588.g4aae82f0.el9.x8 8.6 MB/s | 3.2 MB 00:00 2024-07-26T06:11:02.470 INFO:teuthology.orchestra.run.smithi164.stdout:(79/160): postgresql-server-13.14-1.el9.x86_64. 9.9 MB/s | 5.8 MB 00:00 2024-07-26T06:11:02.522 INFO:teuthology.orchestra.run.smithi164.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-07-26T06:11:02.546 INFO:teuthology.orchestra.run.smithi164.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 467 kB/s | 35 kB 00:00 2024-07-26T06:11:02.550 INFO:teuthology.orchestra.run.smithi163.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 353 kB/s | 41 kB 00:00 2024-07-26T06:11:02.596 INFO:teuthology.orchestra.run.smithi164.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.0 MB/s | 250 kB 00:00 2024-07-26T06:11:02.700 INFO:teuthology.orchestra.run.smithi163.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.6 MB/s | 247 kB 00:00 2024-07-26T06:11:02.721 INFO:teuthology.orchestra.run.smithi164.stdout:(83/160): python3-babel-2.9.1-2.el9.noarch.rpm 8.9 MB/s | 6.0 MB 00:00 2024-07-26T06:11:02.755 INFO:teuthology.orchestra.run.smithi164.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-07-26T06:11:02.758 INFO:teuthology.orchestra.run.smithi163.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 640 kB/s | 37 kB 00:00 2024-07-26T06:11:02.780 INFO:teuthology.orchestra.run.smithi164.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-07-26T06:11:02.805 INFO:teuthology.orchestra.run.smithi164.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 855 kB/s | 42 kB 00:00 2024-07-26T06:11:02.817 INFO:teuthology.orchestra.run.smithi163.stdout:(154/160): elfutils-libelf-0.191-4.el9.x86_64.r 3.5 MB/s | 207 kB 00:00 2024-07-26T06:11:02.839 INFO:teuthology.orchestra.run.smithi164.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-07-26T06:11:02.873 INFO:teuthology.orchestra.run.smithi164.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-07-26T06:11:02.876 INFO:teuthology.orchestra.run.smithi163.stdout:(155/160): NetworkManager-libnm-1.48.2-2.el9.x8 2.9 MB/s | 1.8 MB 00:00 2024-07-26T06:11:02.906 INFO:teuthology.orchestra.run.smithi164.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-07-26T06:11:02.909 INFO:teuthology.orchestra.run.smithi163.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.8 MB/s | 260 kB 00:00 2024-07-26T06:11:02.932 INFO:teuthology.orchestra.run.smithi164.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 883 kB/s | 51 kB 00:00 2024-07-26T06:11:02.957 INFO:teuthology.orchestra.run.smithi164.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-07-26T06:11:02.999 INFO:teuthology.orchestra.run.smithi164.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-07-26T06:11:03.041 INFO:teuthology.orchestra.run.smithi164.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.1 MB/s | 44 kB 00:00 2024-07-26T06:11:03.059 INFO:teuthology.orchestra.run.smithi163.stdout:(157/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.6 MB/s | 2.3 MB 00:00 2024-07-26T06:11:03.091 INFO:teuthology.orchestra.run.smithi164.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-07-26T06:11:03.200 INFO:teuthology.orchestra.run.smithi164.stdout:(95/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 5.1 MB 00:00 2024-07-26T06:11:03.225 INFO:teuthology.orchestra.run.smithi164.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 270 kB/s | 36 kB 00:00 2024-07-26T06:11:03.250 INFO:teuthology.orchestra.run.smithi164.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 959 kB/s | 47 kB 00:00 2024-07-26T06:11:03.284 INFO:teuthology.orchestra.run.smithi164.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-07-26T06:11:03.309 INFO:teuthology.orchestra.run.smithi164.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 996 kB/s | 58 kB 00:00 2024-07-26T06:11:03.366 INFO:teuthology.orchestra.run.smithi164.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.2 MB/s | 96 kB 00:00 2024-07-26T06:11:03.393 INFO:teuthology.orchestra.run.smithi164.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 769 kB/s | 64 kB 00:00 2024-07-26T06:11:03.419 INFO:teuthology.orchestra.run.smithi164.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 867 kB/s | 45 kB 00:00 2024-07-26T06:11:03.527 INFO:teuthology.orchestra.run.smithi164.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 168 kB/s | 22 kB 00:00 2024-07-26T06:11:03.585 INFO:teuthology.orchestra.run.smithi163.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 1.9 MB/s | 1.4 MB 00:00 2024-07-26T06:11:03.627 INFO:teuthology.orchestra.run.smithi163.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 1.9 MB/s | 1.1 MB 00:00 2024-07-26T06:11:03.627 INFO:teuthology.orchestra.run.smithi164.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 5.4 MB/s | 548 kB 00:00 2024-07-26T06:11:03.661 INFO:teuthology.orchestra.run.smithi164.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.2 MB/s | 309 kB 00:00 2024-07-26T06:11:03.678 INFO:teuthology.orchestra.run.smithi164.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-07-26T06:11:03.735 INFO:teuthology.orchestra.run.smithi163.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.9 MB/s | 2.4 MB 00:00 2024-07-26T06:11:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-07-26T06:11:03.745 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:11:03.745 INFO:teuthology.orchestra.run.smithi163.stdout:Total 17 MB/s | 239 MB 00:13 2024-07-26T06:11:04.012 INFO:teuthology.orchestra.run.smithi164.stdout:(108/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-07-26T06:11:04.087 INFO:teuthology.orchestra.run.smithi164.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 74 kB/s | 26 kB 00:00 2024-07-26T06:11:04.121 INFO:teuthology.orchestra.run.smithi164.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 468 kB/s | 49 kB 00:00 2024-07-26T06:11:04.146 INFO:teuthology.orchestra.run.smithi164.stdout:(111/160): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-07-26T06:11:04.205 INFO:teuthology.orchestra.run.smithi164.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-07-26T06:11:04.272 INFO:teuthology.orchestra.run.smithi164.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-07-26T06:11:04.297 INFO:teuthology.orchestra.run.smithi164.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T06:11:04.322 INFO:teuthology.orchestra.run.smithi164.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T06:11:04.356 INFO:teuthology.orchestra.run.smithi164.stdout:(116/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-07-26T06:11:04.381 INFO:teuthology.orchestra.run.smithi164.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 244 kB/s | 14 kB 00:00 2024-07-26T06:11:04.423 INFO:teuthology.orchestra.run.smithi164.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-07-26T06:11:04.465 INFO:teuthology.orchestra.run.smithi164.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-07-26T06:11:04.499 INFO:teuthology.orchestra.run.smithi164.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-07-26T06:11:04.532 INFO:teuthology.orchestra.run.smithi164.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.2 MB/s | 216 kB 00:00 2024-07-26T06:11:04.566 INFO:teuthology.orchestra.run.smithi164.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T06:11:04.591 INFO:teuthology.orchestra.run.smithi164.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 443 kB/s | 11 kB 00:00 2024-07-26T06:11:04.617 INFO:teuthology.orchestra.run.smithi164.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 730 kB/s | 18 kB 00:00 2024-07-26T06:11:04.642 INFO:teuthology.orchestra.run.smithi164.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 955 kB/s | 23 kB 00:00 2024-07-26T06:11:04.667 INFO:teuthology.orchestra.run.smithi164.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 796 kB/s | 19 kB 00:00 2024-07-26T06:11:04.692 INFO:teuthology.orchestra.run.smithi164.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 812 kB/s | 20 kB 00:00 2024-07-26T06:11:04.793 INFO:teuthology.orchestra.run.smithi164.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-07-26T06:11:04.818 INFO:teuthology.orchestra.run.smithi164.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-07-26T06:11:04.844 INFO:teuthology.orchestra.run.smithi164.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.2 MB/s | 79 kB 00:00 2024-07-26T06:11:04.877 INFO:teuthology.orchestra.run.smithi164.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-07-26T06:11:04.911 INFO:teuthology.orchestra.run.smithi164.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-07-26T06:11:04.944 INFO:teuthology.orchestra.run.smithi164.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 505 kB/s | 16 kB 00:00 2024-07-26T06:11:05.029 INFO:teuthology.orchestra.run.smithi164.stdout:(134/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.9 MB/s | 2.0 MB 00:00 2024-07-26T06:11:05.062 INFO:teuthology.orchestra.run.smithi164.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 881 kB/s | 28 kB 00:00 2024-07-26T06:11:05.087 INFO:teuthology.orchestra.run.smithi164.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 635 kB/s | 90 kB 00:00 2024-07-26T06:11:05.187 INFO:teuthology.orchestra.run.smithi164.stdout:(137/160): libarrow-9.0.0-11.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-07-26T06:11:05.213 INFO:teuthology.orchestra.run.smithi164.stdout:(138/160): python3-repoze-lru-0.7-16.el9.noarch 205 kB/s | 31 kB 00:00 2024-07-26T06:11:05.238 INFO:teuthology.orchestra.run.smithi164.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-07-26T06:11:05.263 INFO:teuthology.orchestra.run.smithi164.stdout:(140/160): python3-rsa-4.9-2.el9.noarch.rpm 791 kB/s | 59 kB 00:00 2024-07-26T06:11:05.288 INFO:teuthology.orchestra.run.smithi164.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T06:11:05.322 INFO:teuthology.orchestra.run.smithi164.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 806 kB/s | 188 kB 00:00 2024-07-26T06:11:05.347 INFO:teuthology.orchestra.run.smithi164.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-07-26T06:11:05.364 INFO:teuthology.orchestra.run.smithi164.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-07-26T06:11:05.399 INFO:teuthology.orchestra.run.smithi164.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-07-26T06:11:05.432 INFO:teuthology.orchestra.run.smithi164.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-07-26T06:11:05.473 INFO:teuthology.orchestra.run.smithi164.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-07-26T06:11:05.557 INFO:teuthology.orchestra.run.smithi164.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-07-26T06:11:05.733 INFO:teuthology.orchestra.run.smithi164.stdout:(149/160): librados2-19.0.0-4588.g4aae82f0.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T06:11:05.808 INFO:teuthology.orchestra.run.smithi164.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 9.3 MB/s | 2.3 MB 00:00 2024-07-26T06:11:05.891 INFO:teuthology.orchestra.run.smithi164.stdout:(151/160): librbd1-19.0.0-4588.g4aae82f0.el9.x8 7.6 MB/s | 3.2 MB 00:00 2024-07-26T06:11:05.917 INFO:teuthology.orchestra.run.smithi164.stdout:(152/160): NetworkManager-team-1.48.2-2.el9.x86 378 kB/s | 41 kB 00:00 2024-07-26T06:11:05.984 INFO:teuthology.orchestra.run.smithi164.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 558 kB/s | 37 kB 00:00 2024-07-26T06:11:06.034 INFO:teuthology.orchestra.run.smithi164.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.7 MB/s | 247 kB 00:00 2024-07-26T06:11:06.093 INFO:teuthology.orchestra.run.smithi164.stdout:(155/160): NetworkManager-libnm-1.48.2-2.el9.x8 5.0 MB/s | 1.8 MB 00:00 2024-07-26T06:11:06.135 INFO:teuthology.orchestra.run.smithi164.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.3 MB/s | 207 kB 00:00 2024-07-26T06:11:06.185 INFO:teuthology.orchestra.run.smithi164.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.7 MB/s | 260 kB 00:00 2024-07-26T06:11:06.327 INFO:teuthology.orchestra.run.smithi164.stdout:(158/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-07-26T06:11:06.377 INFO:teuthology.orchestra.run.smithi164.stdout:(159/160): openssl-3.2.2-2.el9.x86_64.rpm 4.8 MB/s | 1.4 MB 00:00 2024-07-26T06:11:06.435 INFO:teuthology.orchestra.run.smithi164.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 4.3 MB/s | 1.1 MB 00:00 2024-07-26T06:11:06.448 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:11:06.448 INFO:teuthology.orchestra.run.smithi164.stdout:Total 21 MB/s | 239 MB 00:11 2024-07-26T06:11:06.934 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-07-26T06:11:07.219 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-07-26T06:11:07.219 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-07-26T06:11:09.764 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-07-26T06:11:09.926 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-07-26T06:11:09.927 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-07-26T06:11:10.057 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-07-26T06:11:10.057 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-07-26T06:11:12.747 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-07-26T06:11:12.748 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-07-26T06:11:16.931 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-07-26T06:11:17.040 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T06:11:17.127 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T06:11:17.257 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T06:11:17.312 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T06:11:17.343 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T06:11:17.450 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T06:11:17.882 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T06:11:17.982 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 8/172 2024-07-26T06:11:18.159 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 8/172 2024-07-26T06:11:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 9/172 2024-07-26T06:11:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 9/172 2024-07-26T06:11:18.366 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-4588.g4aae82f0.el9.x86_64 10/172 2024-07-26T06:11:18.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T06:11:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T06:11:18.796 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T06:11:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 14/172 2024-07-26T06:11:18.957 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 14/172 2024-07-26T06:11:18.997 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86 15/172 2024-07-26T06:11:19.075 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86 15/172 2024-07-26T06:11:19.107 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:11:19.169 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:11:19.472 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-07-26T06:11:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:11:19.558 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T06:11:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T06:11:19.659 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:11:19.699 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T06:11:19.748 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T06:11:19.784 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T06:11:19.889 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T06:11:20.053 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T06:11:20.240 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:11:20.283 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T06:11:20.320 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:11:20.383 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 8/172 2024-07-26T06:11:20.514 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:11:20.556 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 8/172 2024-07-26T06:11:20.592 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 9/172 2024-07-26T06:11:20.618 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:11:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 9/172 2024-07-26T06:11:20.776 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-4588.g4aae82f0.el9.x86_64 10/172 2024-07-26T06:11:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T06:11:20.969 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:11:21.136 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T06:11:21.173 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T06:11:21.185 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T06:11:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T06:11:21.253 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T06:11:21.280 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 14/172 2024-07-26T06:11:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T06:11:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T06:11:21.444 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 14/172 2024-07-26T06:11:21.445 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T06:11:21.473 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86 15/172 2024-07-26T06:11:21.507 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T06:11:21.556 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T06:11:21.559 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86 15/172 2024-07-26T06:11:21.589 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T06:11:21.591 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:11:21.670 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:11:21.720 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T06:11:21.795 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T06:11:21.833 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T06:11:21.889 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T06:11:21.973 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:11:21.998 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T06:11:22.038 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-4588.g4aae82f0.el9. 35/172 2024-07-26T06:11:22.086 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-4588.g4aae82f0.el 36/172 2024-07-26T06:11:22.119 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-4588.g4aae82f0.el9.x86_6 37/172 2024-07-26T06:11:22.133 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:11:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T06:11:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T06:11:22.491 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T06:11:22.516 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T06:11:22.699 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:11:22.732 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T06:11:22.790 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:11:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T06:11:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:11:23.096 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:11:23.478 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T06:11:23.574 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T06:11:23.635 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T06:11:23.674 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T06:11:23.713 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:11:23.740 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T06:11:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-4588.g4aae82f0.el9.x86_6 48/172 2024-07-26T06:11:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T06:11:24.069 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T06:11:24.109 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T06:11:24.111 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T06:11:24.167 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T06:11:24.180 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T06:11:24.289 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T06:11:24.313 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T06:11:24.341 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T06:11:24.394 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T06:11:24.396 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T06:11:24.459 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T06:11:24.479 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T06:11:24.509 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T06:11:24.542 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T06:11:24.551 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T06:11:24.577 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T06:11:24.648 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T06:11:24.662 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T06:11:24.695 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T06:11:24.728 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T06:11:24.758 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T06:11:24.776 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T06:11:24.833 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T06:11:24.839 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T06:11:24.943 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T06:11:24.981 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-4588.g4aae82f0.el9. 35/172 2024-07-26T06:11:25.030 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T06:11:25.032 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-4588.g4aae82f0.el 36/172 2024-07-26T06:11:25.061 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-4588.g4aae82f0.el9.x86_6 37/172 2024-07-26T06:11:25.100 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T06:11:25.160 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T06:11:25.172 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T06:11:25.232 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T06:11:25.294 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T06:11:25.332 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T06:11:25.345 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T06:11:25.399 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T06:11:25.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T06:11:25.469 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T06:11:25.505 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T06:11:25.567 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T06:11:25.657 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T06:11:25.839 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T06:11:25.883 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T06:11:25.913 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T06:11:25.964 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T06:11:26.013 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T06:11:26.060 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T06:11:26.102 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T06:11:26.147 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T06:11:26.189 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T06:11:26.249 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T06:11:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T06:11:26.339 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T06:11:26.348 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T06:11:26.386 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T06:11:26.432 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T06:11:26.480 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T06:11:26.496 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T06:11:26.536 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T06:11:26.541 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-4588.g4aae82f0.el9.x86_6 48/172 2024-07-26T06:11:26.580 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T06:11:26.617 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T06:11:26.718 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T06:11:26.784 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:11:26.830 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T06:11:26.930 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T06:11:27.063 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T06:11:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T06:11:27.226 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T06:11:27.271 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T06:11:27.310 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T06:11:27.396 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T06:11:27.437 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T06:11:27.499 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T06:11:27.564 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T06:11:27.749 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T06:11:27.762 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T06:11:27.808 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T06:11:27.820 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T06:11:27.841 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T06:11:27.866 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T06:11:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T06:11:27.995 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T06:11:28.024 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T06:11:28.036 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T06:11:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T06:11:28.088 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T06:11:28.127 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T06:11:28.161 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T06:11:28.187 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T06:11:28.235 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T06:11:28.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T06:11:28.457 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T06:11:28.490 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T06:11:28.540 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T06:11:28.592 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T06:11:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T06:11:28.629 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T06:11:28.669 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T06:11:28.723 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T06:11:28.778 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T06:11:28.819 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T06:11:28.865 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T06:11:28.924 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T06:11:28.999 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T06:11:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T06:11:29.187 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T06:11:29.262 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T06:11:29.305 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T06:11:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T06:11:29.476 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T06:11:29.541 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:11:29.577 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T06:11:30.504 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T06:11:30.558 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T06:11:30.591 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T06:11:30.712 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T06:11:30.774 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T06:11:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T06:11:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T06:11:31.002 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T06:11:31.192 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T06:11:31.228 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T06:11:31.265 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T06:11:31.304 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T06:11:31.339 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T06:11:31.370 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T06:11:31.377 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T06:11:31.775 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T06:11:31.816 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 104/172 2024-07-26T06:11:31.975 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 104/172 2024-07-26T06:11:32.024 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 105/172 2024-07-26T06:11:32.081 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T06:11:32.187 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T06:11:32.329 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T06:11:32.667 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T06:11:32.694 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 108/172 2024-07-26T06:11:32.796 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 108/172 2024-07-26T06:11:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 109/172 2024-07-26T06:11:33.940 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T06:11:34.037 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T06:11:34.072 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T06:11:34.107 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T06:11:34.467 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T06:11:34.508 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 104/172 2024-07-26T06:11:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 104/172 2024-07-26T06:11:34.666 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 105/172 2024-07-26T06:11:34.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T06:11:35.282 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T06:11:35.311 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 108/172 2024-07-26T06:11:35.412 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 108/172 2024-07-26T06:11:35.438 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 109/172 2024-07-26T06:11:35.496 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 110/172 2024-07-26T06:11:35.534 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 110/172 2024-07-26T06:11:36.182 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 110/172 2024-07-26T06:11:36.229 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 111/172 2024-07-26T06:11:36.388 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 111/172 2024-07-26T06:11:36.389 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T06:11:36.389 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T06:11:36.389 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:36.438 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 112/172 2024-07-26T06:11:38.075 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 110/172 2024-07-26T06:11:38.117 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 110/172 2024-07-26T06:11:38.803 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 110/172 2024-07-26T06:11:38.854 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 111/172 2024-07-26T06:11:39.002 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 111/172 2024-07-26T06:11:39.002 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T06:11:39.002 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T06:11:39.003 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:11:39.047 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 112/172 2024-07-26T06:11:49.189 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 112/172 2024-07-26T06:11:49.189 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-07-26T06:11:49.189 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-07-26T06:11:49.190 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-07-26T06:11:49.190 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-07-26T06:11:49.190 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-07-26T06:11:49.190 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-07-26T06:11:49.190 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-07-26T06:11:49.190 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 113/172 2024-07-26T06:11:49.481 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 113/172 2024-07-26T06:11:49.482 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:49.482 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T06:11:49.482 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:51.119 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 114/172 2024-07-26T06:11:51.197 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 114/172 2024-07-26T06:11:51.198 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:51.198 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T06:11:51.198 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:11:51.198 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:11:51.198 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:51.469 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 115/172 2024-07-26T06:11:51.540 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 115/172 2024-07-26T06:11:51.540 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:51.540 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T06:11:51.540 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:11:51.540 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:11:51.540 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:51.703 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 112/172 2024-07-26T06:11:51.704 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-07-26T06:11:51.704 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-07-26T06:11:51.704 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-07-26T06:11:51.704 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-07-26T06:11:51.704 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-07-26T06:11:51.704 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-07-26T06:11:51.705 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-07-26T06:11:51.705 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:11:51.945 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 113/172 2024-07-26T06:11:51.994 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 113/172 2024-07-26T06:11:51.994 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:51.994 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T06:11:51.994 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:11:52.020 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 116/172 2024-07-26T06:11:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 116/172 2024-07-26T06:11:52.093 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:52.093 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T06:11:52.093 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:11:52.093 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:11:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:52.256 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T06:11:52.320 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T06:11:52.394 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T06:11:52.648 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T06:11:52.718 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T06:11:52.808 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T06:11:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T06:11:53.097 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T06:11:53.150 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T06:11:53.190 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T06:11:53.216 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T06:11:53.623 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 114/172 2024-07-26T06:11:53.701 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 114/172 2024-07-26T06:11:53.701 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:53.701 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T06:11:53.701 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:11:53.702 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:11:53.702 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:11:53.871 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 128/172 2024-07-26T06:11:53.910 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 128/172 2024-07-26T06:11:53.948 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 115/172 2024-07-26T06:11:53.997 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4588.g4aae82f0.e 129/172 2024-07-26T06:11:54.015 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 115/172 2024-07-26T06:11:54.015 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:54.015 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T06:11:54.015 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:11:54.015 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:11:54.015 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:11:54.111 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4588.g4aae82f0. 130/172 2024-07-26T06:11:54.147 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noa 131/172 2024-07-26T06:11:54.507 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 116/172 2024-07-26T06:11:54.578 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 116/172 2024-07-26T06:11:54.578 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:54.578 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T06:11:54.578 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:11:54.579 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:11:54.579 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:11:54.707 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T06:11:54.770 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T06:11:54.842 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T06:11:55.068 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T06:11:55.136 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T06:11:55.231 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T06:11:55.339 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T06:11:55.503 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T06:11:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T06:11:55.592 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T06:11:55.626 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T06:11:56.338 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 128/172 2024-07-26T06:11:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 128/172 2024-07-26T06:11:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4588.g4aae82f0.e 129/172 2024-07-26T06:11:56.522 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4588.g4aae82f0. 130/172 2024-07-26T06:11:56.548 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noa 131/172 2024-07-26T06:11:57.790 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noa 131/172 2024-07-26T06:11:57.854 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.n 132/172 2024-07-26T06:11:58.944 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.n 132/172 2024-07-26T06:11:58.970 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aa 133/172 2024-07-26T06:11:59.126 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aa 133/172 2024-07-26T06:11:59.331 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4588.g4aae82f0.el 134/172 2024-07-26T06:11:59.364 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 135/172 2024-07-26T06:11:59.435 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 135/172 2024-07-26T06:11:59.435 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:11:59.435 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T06:11:59.435 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:11:59.435 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:11:59.435 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:11:59.499 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 136/172 2024-07-26T06:11:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 136/172 2024-07-26T06:11:59.687 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-4588.g4aae82f0.el9.x86_64 137/172 2024-07-26T06:11:59.794 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 138/172 2024-07-26T06:12:00.180 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noa 131/172 2024-07-26T06:12:00.247 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.n 132/172 2024-07-26T06:12:00.867 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T06:12:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 140/172 2024-07-26T06:12:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 140/172 2024-07-26T06:12:00.969 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:00.969 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T06:12:00.969 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:12:00.969 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:12:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:01.027 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4588.g4aae8 141/172 2024-07-26T06:12:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4588.g4aae8 141/172 2024-07-26T06:12:01.097 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:01.098 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T06:12:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:01.170 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 142/172 2024-07-26T06:12:01.244 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 142/172 2024-07-26T06:12:01.244 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:01.244 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T06:12:01.244 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:12:01.245 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:12:01.245 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:01.341 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.n 132/172 2024-07-26T06:12:01.371 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aa 133/172 2024-07-26T06:12:01.529 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aa 133/172 2024-07-26T06:12:01.595 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 143/172 2024-07-26T06:12:01.664 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 143/172 2024-07-26T06:12:01.665 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:01.665 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T06:12:01.665 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:12:01.665 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:12:01.665 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:01.702 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4588.g4aae82f0.el 134/172 2024-07-26T06:12:01.732 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 135/172 2024-07-26T06:12:01.805 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 135/172 2024-07-26T06:12:01.806 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:01.806 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T06:12:01.806 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:12:01.806 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:12:01.806 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:01.858 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 136/172 2024-07-26T06:12:01.893 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 136/172 2024-07-26T06:12:02.020 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-4588.g4aae82f0.el9.x86_64 137/172 2024-07-26T06:12:02.086 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 138/172 2024-07-26T06:12:03.130 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T06:12:03.169 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 140/172 2024-07-26T06:12:03.239 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 140/172 2024-07-26T06:12:03.240 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:03.240 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T06:12:03.240 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:12:03.240 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:12:03.240 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:03.294 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4588.g4aae8 141/172 2024-07-26T06:12:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4588.g4aae8 141/172 2024-07-26T06:12:03.362 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:03.362 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T06:12:03.362 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:03.420 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 142/172 2024-07-26T06:12:03.513 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 142/172 2024-07-26T06:12:03.513 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:03.513 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T06:12:03.513 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:12:03.513 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:12:03.513 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:03.887 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 143/172 2024-07-26T06:12:03.965 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 143/172 2024-07-26T06:12:03.965 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:12:03.965 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T06:12:03.966 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:12:03.966 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:12:03.966 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:06.651 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-4588.g4aae82f0.el9.x86_64 144/172 2024-07-26T06:12:06.707 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 145/172 2024-07-26T06:12:06.761 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 146/172 2024-07-26T06:12:06.801 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T06:12:06.839 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T06:12:07.123 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-shell-2:19.0.0-4588.g4aae82f0.el9.x86_64 149/172 2024-07-26T06:12:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T06:12:07.230 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-4588.g4aae82f0.el9.x86_ 151/172 2024-07-26T06:12:07.261 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T06:12:07.307 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:12:07.355 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T06:12:08.340 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:12:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:12:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:12:08.955 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-4588.g4aae82f0.el9.x86_64 144/172 2024-07-26T06:12:09.001 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 145/172 2024-07-26T06:12:09.054 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 146/172 2024-07-26T06:12:09.112 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T06:12:09.152 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T06:12:09.435 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-shell-2:19.0.0-4588.g4aae82f0.el9.x86_64 149/172 2024-07-26T06:12:09.484 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T06:12:09.516 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-4588.g4aae82f0.el9.x86_ 151/172 2024-07-26T06:12:09.548 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T06:12:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:12:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T06:12:09.835 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-top-2:19.0.0-4588.g4aae82f0.el9.noarch 156/172 2024-07-26T06:12:09.970 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T06:12:10.034 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T06:12:10.199 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T06:12:10.238 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T06:12:10.272 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T06:12:10.308 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:12:10.349 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T06:12:10.349 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:12:10.465 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:12:10.512 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T06:12:10.512 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:12:10.532 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:12:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:12:10.641 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:12:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:12:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T06:12:10.755 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:12:10.756 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:12:11.096 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:12:11.097 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:12:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:12:11.315 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:12:11.364 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T06:12:11.502 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:12:11.769 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:12:11.769 INFO:teuthology.orchestra.run.smithi163.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T06:12:11.769 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:12.016 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-top-2:19.0.0-4588.g4aae82f0.el9.noarch 156/172 2024-07-26T06:12:12.115 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T06:12:12.175 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T06:12:12.314 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T06:12:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T06:12:12.416 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T06:12:12.450 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:12:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T06:12:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:12:12.633 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:12:12.683 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T06:12:12.683 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:12:12.812 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:12:12.851 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T06:12:12.884 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:12:12.885 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:12:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:12:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:12:13.410 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:12:13.455 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:12:13.502 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T06:12:13.631 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:12:13.893 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:12:13.893 INFO:teuthology.orchestra.run.smithi164.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T06:12:13.893 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:18.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-4588.g4aae82f0.el9.x86_64 1/172 2024-07-26T06:12:18.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 2/172 2024-07-26T06:12:18.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 3/172 2024-07-26T06:12:18.174 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 4/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4588.g4aae8 5/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 6/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 7/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 8/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 9/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 10/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 11/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-4588.g4aae82f0.el9.x86_64 12/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 13/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-shell-2:19.0.0-4588.g4aae82f0.el9.x86_64 14/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-4588.g4aae82f0.el9.x86_ 15/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 16/172 2024-07-26T06:12:18.175 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 17/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-4588.g4aae82f0.el9.x86_6 18/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86 19/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 20/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4588.g4aae82f0.el 21/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-4588.g4aae82f0.el9. 22/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-4588.g4aae82f0.el9.x86_6 23/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-4588.g4aae82f0.el9.x86_64 24/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 25/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 26/172 2024-07-26T06:12:18.176 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 27/172 2024-07-26T06:12:18.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 28/172 2024-07-26T06:12:18.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 29/172 2024-07-26T06:12:18.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4588.g4aae82f0. 30/172 2024-07-26T06:12:18.189 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noa 31/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.n 32/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aa 33/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4588.g4aae82f0.el 34/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 35/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4588.g4aae82f0.e 36/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 37/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 38/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-top-2:19.0.0-4588.g4aae82f0.el9.noarch 39/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T06:12:18.190 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T06:12:18.191 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T06:12:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T06:12:18.193 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T06:12:18.194 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T06:12:18.195 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T06:12:18.196 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T06:12:18.197 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T06:12:18.198 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 149/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 151/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T06:12:18.199 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:12:18.200 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T06:12:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-4588.g4aae82f0.el9.x86_64 1/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 2/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 3/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 4/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4588.g4aae8 5/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 6/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 7/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 8/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 9/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 10/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 11/172 2024-07-26T06:12:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-4588.g4aae82f0.el9.x86_64 12/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 13/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-shell-2:19.0.0-4588.g4aae82f0.el9.x86_64 14/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-4588.g4aae82f0.el9.x86_ 15/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 16/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 17/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-4588.g4aae82f0.el9.x86_6 18/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86 19/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 20/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4588.g4aae82f0.el 21/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-4588.g4aae82f0.el9. 22/172 2024-07-26T06:12:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-4588.g4aae82f0.el9.x86_6 23/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-4588.g4aae82f0.el9.x86_64 24/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 25/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 26/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 27/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 28/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 29/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4588.g4aae82f0. 30/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noa 31/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.n 32/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aa 33/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4588.g4aae82f0.el 34/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 35/172 2024-07-26T06:12:19.933 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4588.g4aae82f0.e 36/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 37/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 38/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-top-2:19.0.0-4588.g4aae82f0.el9.noarch 39/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T06:12:19.934 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T06:12:19.935 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T06:12:19.936 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T06:12:19.937 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T06:12:19.938 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T06:12:19.939 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T06:12:19.940 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T06:12:19.941 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T06:12:19.942 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 149/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 151/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T06:12:19.943 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:12:19.944 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:20.570 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T06:12:20.571 INFO:teuthology.orchestra.run.smithi163.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.572 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.573 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T06:12:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T06:12:20.575 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T06:12:20.576 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T06:12:20.577 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T06:12:20.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T06:12:20.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T06:12:20.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T06:12:20.579 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T06:12:20.580 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T06:12:20.581 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T06:12:20.582 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.583 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:20.584 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-07-26T06:12:21.263 DEBUG:teuthology.parallel:result is None 2024-07-26T06:12:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:12:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:22.245 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T06:12:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.247 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.248 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top-2:19.0.0-4588.g4aae82f0.el9.noarch 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T06:12:22.249 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.250 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T06:12:22.251 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T06:12:22.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T06:12:22.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T06:12:22.254 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T06:12:22.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T06:12:22.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T06:12:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-4588.g4aae82f0.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T06:12:22.258 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T06:12:22.259 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T06:12:22.259 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:22.259 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-07-26T06:12:22.875 DEBUG:teuthology.parallel:result is None 2024-07-26T06:12:22.876 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4aae82f07003420ccc152fe0b01f3d5a44885708 2024-07-26T06:12:23.026 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T06:12:23.079 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-4588.g4aae82f0.el9 2024-07-26T06:12:23.080 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4588.g4aae82f0.el9 2024-07-26T06:12:23.080 INFO:teuthology.task.install:The correct ceph version 19.0.0-4588.g4aae82f0 is installed. 2024-07-26T06:12:23.082 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4aae82f07003420ccc152fe0b01f3d5a44885708 2024-07-26T06:12:23.218 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T06:12:23.269 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-4588.g4aae82f0.el9 2024-07-26T06:12:23.269 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4588.g4aae82f0.el9 2024-07-26T06:12:23.269 INFO:teuthology.task.install:The correct ceph version 19.0.0-4588.g4aae82f0 is installed. 2024-07-26T06:12:23.271 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T06:12:23.272 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:23.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T06:12:23.337 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:23.337 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T06:12:23.402 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T06:12:23.403 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:23.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T06:12:23.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T06:12:23.583 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:23.583 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T06:12:23.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T06:12:23.753 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T06:12:23.754 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:23.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T06:12:23.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T06:12:23.932 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:23.932 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T06:12:23.993 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T06:12:24.094 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T06:12:24.095 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:24.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T06:12:24.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T06:12:24.271 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:24.271 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T06:12:24.341 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T06:12:24.445 INFO:teuthology.run_tasks:Running task ceph... 2024-07-26T06:12:24.525 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-07-26T06:12:24.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /var/log/ceph 2024-07-26T06:12:24.532 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-07-26T06:12:24.609 INFO:tasks.ceph:Disabling ceph logrotate... 2024-07-26T06:12:24.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/logrotate.d/ceph 2024-07-26T06:12:24.614 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-07-26T06:12:24.687 INFO:tasks.ceph:Creating extra log directories... 2024-07-26T06:12:24.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-07-26T06:12:24.692 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-07-26T06:12:24.795 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-07-26T06:12:24.795 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-07-26T06:12:24.795 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-07-26T06:12:24.796 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:24.796 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-07-26T06:12:24.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-07-26T06:12:25.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-07-26T06:12:25.156 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:25.156 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-07-26T06:12:25.199 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-07-26T06:12:25.376 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-07-26T06:12:25.462 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-07-26T06:12:25.462 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-07-26T06:12:25.462 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960', 'branch': 'rishabh-mgr-vol-clone-stats-bugfixes', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}}', 'email': None, 'first_in_suite': False, 'job_id': '7818960', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-mgr-vol-clone-stats-bugfixes'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-mgr-vol-clone-stats-bugfixes', 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6921, 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-mgr-vol-clone-stats-bugfixes', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi163.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJPgve1quZVfhUSvWDNm7uSDkV8/n1fQNdG2XGx19Q/L0XaeVA4kQADjwEAzvpiP+Kn8Lh4ySQlH+QmZfHFBTJY=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC+emzptfrRxxWzFEY3+r2b+TICzh8x5GxHKRxKsmq39jz8M93Os7XJC7FgvlB2CPfwqb+gk0lAwQVLgXuY1RnY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4aae82f07003420ccc152fe0b01f3d5a44885708', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestMisc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '53ce1462e129f6eb4071986336534c740fdebd31', 'timestamp': '2024-07-26_05:39:42', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525'} 2024-07-26T06:12:25.463 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-07-26T06:12:25.471 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-07-26T06:12:25.518 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/run/ceph 2024-07-26T06:12:25.523 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-07-26T06:12:25.617 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:25.617 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T06:12:25.669 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T06:12:25.669 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-07-26T06:12:25.720 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T06:12:25.720 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:25.720 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-07-26T06:12:25.720 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:25.720 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:25.720 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:12:12.230378722 +0000 2024-07-26T06:12:25.721 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:09:15.617767411 +0000 2024-07-26T06:12:25.721 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:09:15.617767411 +0000 2024-07-26T06:12:25.721 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:09:15.617767411 +0000 2024-07-26T06:12:25.721 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T06:12:25.802 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:12:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:12:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000381933 s, 1.3 MB/s 2024-07-26T06:12:25.806 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T06:12:25.864 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-07-26T06:12:25.914 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T06:12:25.914 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:25.914 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-07-26T06:12:25.914 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:25.914 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:25.915 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:12:12.231378699 +0000 2024-07-26T06:12:25.915 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:09:16.181755177 +0000 2024-07-26T06:12:25.915 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:09:16.182755155 +0000 2024-07-26T06:12:25.915 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:09:16.181755177 +0000 2024-07-26T06:12:25.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T06:12:25.990 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:12:25.991 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:12:25.991 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000362885 s, 1.4 MB/s 2024-07-26T06:12:25.993 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T06:12:26.048 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-07-26T06:12:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T06:12:26.096 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:12:12.232378677 +0000 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:09:16.780742183 +0000 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:09:16.780742183 +0000 2024-07-26T06:12:26.097 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:09:16.780742183 +0000 2024-07-26T06:12:26.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T06:12:26.174 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:12:26.174 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:12:26.174 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000327188 s, 1.6 MB/s 2024-07-26T06:12:26.177 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T06:12:26.232 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-07-26T06:12:26.280 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:12:12.233378655 +0000 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:09:17.377729233 +0000 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:09:17.377729233 +0000 2024-07-26T06:12:26.281 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:09:17.377729233 +0000 2024-07-26T06:12:26.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T06:12:26.360 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:12:26.360 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:12:26.360 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000372532 s, 1.4 MB/s 2024-07-26T06:12:26.364 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T06:12:26.419 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:26.420 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:26.420 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T06:12:26.478 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T06:12:26.478 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-07-26T06:12:26.528 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T06:12:26.528 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:12:14.335657748 +0000 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:09:19.827599908 +0000 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:09:19.827599908 +0000 2024-07-26T06:12:26.529 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:09:19.827599908 +0000 2024-07-26T06:12:26.529 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T06:12:26.607 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:12:26.607 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:12:26.607 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000418496 s, 1.2 MB/s 2024-07-26T06:12:26.611 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T06:12:26.668 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:12:14.335657748 +0000 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:09:20.423590000 +0000 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:09:20.423590000 +0000 2024-07-26T06:12:26.719 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:09:20.423590000 +0000 2024-07-26T06:12:26.720 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T06:12:26.792 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:12:26.792 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:12:26.792 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00030214 s, 1.7 MB/s 2024-07-26T06:12:26.795 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T06:12:26.851 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-07-26T06:12:26.902 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T06:12:26.902 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:26.902 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-07-26T06:12:26.903 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:26.903 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:26.903 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:12:14.336657732 +0000 2024-07-26T06:12:26.903 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:09:21.016580142 +0000 2024-07-26T06:12:26.903 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:09:21.016580142 +0000 2024-07-26T06:12:26.903 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:09:21.016580142 +0000 2024-07-26T06:12:26.903 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T06:12:26.980 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:12:26.980 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:12:26.980 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000361618 s, 1.4 MB/s 2024-07-26T06:12:26.983 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T06:12:27.036 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-07-26T06:12:27.080 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T06:12:27.080 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:12:27.080 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-07-26T06:12:27.080 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:12:27.080 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:12:27.081 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:12:14.336657732 +0000 2024-07-26T06:12:27.081 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:09:21.652569568 +0000 2024-07-26T06:12:27.081 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:09:21.652569568 +0000 2024-07-26T06:12:27.081 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:09:21.652569568 +0000 2024-07-26T06:12:27.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T06:12:27.196 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:12:27.196 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:12:27.196 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00037051 s, 1.4 MB/s 2024-07-26T06:12:27.199 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T06:12:27.251 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:27.252 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-07-26T06:12:27.252 INFO:tasks.ceph:Generating config... 2024-07-26T06:12:27.253 INFO:tasks.ceph:[client] pid file = 2024-07-26T06:12:27.253 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-07-26T06:12:27.253 INFO:tasks.ceph:[mgr] debug client = 20 2024-07-26T06:12:27.253 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-07-26T06:12:27.253 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-07-26T06:12:27.253 INFO:tasks.ceph:[mgr] debug ms = 1 2024-07-26T06:12:27.253 INFO:tasks.ceph:[mon] debug mon = 20 2024-07-26T06:12:27.254 INFO:tasks.ceph:[mon] debug ms = 1 2024-07-26T06:12:27.254 INFO:tasks.ceph:[mon] debug paxos = 20 2024-07-26T06:12:27.254 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-07-26T06:12:27.254 INFO:tasks.ceph:[osd] bdev async discard = True 2024-07-26T06:12:27.254 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-07-26T06:12:27.254 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-07-26T06:12:27.254 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] debug ms = 1 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] debug osd = 20 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-07-26T06:12:27.255 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-07-26T06:12:27.255 INFO:tasks.ceph:Setting up mon.a... 2024-07-26T06:12:27.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-07-26T06:12:27.386 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.keyring 2024-07-26T06:12:27.394 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-07-26T06:12:27.518 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-26T06:12:27.598 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.163'), ('mon.b', '172.21.15.164'), ('mon.c', '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]')] 2024-07-26T06:12:27.599 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.163,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-07-26T06:12:27.600 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:27.600 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-07-26T06:12:27.653 DEBUG:teuthology.orchestra.run.smithi163:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.163 --add b 172.21.15.164 --addv c '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:27.758 INFO:teuthology.orchestra.run.smithi163.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:monmaptool: generated fsid a1639b17-d73f-4cae-9471-025ba01f9572 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:setting min_mon_release = quincy 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:epoch 0 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:fsid a1639b17-d73f-4cae-9471-025ba01f9572 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:last_changed 2024-07-26T06:12:27.759164+0000 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:created 2024-07-26T06:12:27.759164+0000 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:min_mon_release 17 (quincy) 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:election_strategy: 1 2024-07-26T06:12:27.759 INFO:teuthology.orchestra.run.smithi163.stdout:0: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.a 2024-07-26T06:12:27.760 INFO:teuthology.orchestra.run.smithi163.stdout:1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.b 2024-07-26T06:12:27.760 INFO:teuthology.orchestra.run.smithi163.stdout:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-07-26T06:12:27.760 INFO:teuthology.orchestra.run.smithi163.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-07-26T06:12:27.762 DEBUG:teuthology.orchestra.run.smithi163:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-07-26T06:12:27.812 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a1639b17-d73f-4cae-9471-025ba01f9572... 2024-07-26T06:12:27.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-07-26T06:12:27.820 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-07-26T06:12:27.950 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2024-07-26T06:12:27.950 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2024-07-26T06:12:27.950 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.951 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2024-07-26T06:12:27.952 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon_allow_pool_size_one = true 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2024-07-26T06:12:27.953 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.163,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on skipped message = False 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = a1639b17-d73f-4cae-9471-025ba01f9572 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: osd mclock profile = high_recovery_ops 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2024-07-26T06:12:27.954 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug trim objects = true 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.955 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: bdev async discard = True 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: bdev enable discard = True 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore allocator = bitmap 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore block size = 96636764160 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore fsck on mount = True 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluefs = 1/20 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluestore = 1/20 2024-07-26T06:12:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: debug rocksdb = 4/10 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd backfillfull_ratio = 0.85 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd full ratio = 0.9 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd nearfull ratio = 0.8 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: osd failsafe full ratio = 0.95 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = bluestore 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-07-26T06:12:27.957 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: debug client = 20 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: debug finisher = 20 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2024-07-26T06:12:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-07-26T06:12:27.959 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2024-07-26T06:12:27.959 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2024-07-26T06:12:27.959 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-07-26T06:12:27.959 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2024-07-26T06:12:27.959 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2024-07-26T06:12:27.959 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: # 1m isn't quite enough 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: mon_down_mkfs_grace = 2m 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_filestore_osds = false 2024-07-26T06:12:27.960 INFO:teuthology.orchestra.run.smithi163.stdout: mon down mkfs grace = 300 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = "" 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.a] 2024-07-26T06:12:27.961 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.c] 2024-07-26T06:12:27.962 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.b] 2024-07-26T06:12:27.962 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.d] 2024-07-26T06:12:27.963 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-07-26T06:12:27.963 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-07-26T06:12:27.963 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.964 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-07-26T06:12:27.965 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-07-26T06:12:27.966 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.163,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on skipped message = False 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = a1639b17-d73f-4cae-9471-025ba01f9572 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.967 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-07-26T06:12:27.968 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-07-26T06:12:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-07-26T06:12:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-07-26T06:12:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: debug finisher = 20 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-07-26T06:12:27.972 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-07-26T06:12:27.973 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: mon down mkfs grace = 300 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-07-26T06:12:27.974 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-07-26T06:12:27.975 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-07-26T06:12:27.975 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-07-26T06:12:27.975 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-07-26T06:12:27.975 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-07-26T06:12:27.975 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-07-26T06:12:27.975 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-07-26T06:12:27.998 INFO:tasks.ceph:Creating admin key on mon.a... 2024-07-26T06:12:27.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-07-26T06:12:28.138 INFO:tasks.ceph:Copying monmap to all nodes... 2024-07-26T06:12:28.139 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:28.140 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-07-26T06:12:28.190 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:28.190 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-07-26T06:12:28.287 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:12:28.287 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:28.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.keyring 2024-07-26T06:12:28.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-26T06:12:28.401 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:12:28.402 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:28.496 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:12:28.496 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:28.496 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2024-07-26T06:12:28.496 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-07-26T06:12:28.620 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:12:28.620 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:28.717 INFO:tasks.ceph:Setting up mon nodes... 2024-07-26T06:12:28.717 INFO:tasks.ceph:Setting up mgr nodes... 2024-07-26T06:12:28.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-07-26T06:12:28.869 INFO:teuthology.orchestra.run.smithi163.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-07-26T06:12:28.876 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-07-26T06:12:29.042 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-07-26T06:12:29.053 INFO:tasks.ceph:Setting up mds nodes... 2024-07-26T06:12:29.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-07-26T06:12:29.194 INFO:teuthology.orchestra.run.smithi163.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-07-26T06:12:29.201 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-07-26T06:12:29.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-07-26T06:12:29.429 INFO:teuthology.orchestra.run.smithi163.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-07-26T06:12:29.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-07-26T06:12:29.516 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-07-26T06:12:29.671 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-07-26T06:12:29.680 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-07-26T06:12:29.765 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-07-26T06:12:29.907 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-07-26T06:12:29.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-07-26T06:12:29.989 INFO:tasks.ceph_client:Setting up client nodes... 2024-07-26T06:12:29.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T06:12:30.103 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-07-26T06:12:30.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-07-26T06:12:30.257 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-07-26T06:12:30.304 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-07-26T06:12:30.425 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-07-26T06:12:30.469 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-07-26T06:12:30.581 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-07-26T06:12:30.618 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-07-26T06:12:30.618 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-07-26T06:12:30.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-07-26T06:12:30.709 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:30.710 INFO:tasks.ceph:role: osd.0 2024-07-26T06:12:30.710 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:12:30.710 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-07-26T06:12:30.798 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:30.799 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:30.799 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:30.799 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:30.799 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:30.799 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:30.800 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:30.800 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:30.800 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:30.800 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:31.211 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-07-26T06:12:31.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-07-26T06:12:31.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-07-26T06:12:31.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-07-26T06:12:31.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-07-26T06:12:31.541 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:31.541 INFO:tasks.ceph:role: osd.1 2024-07-26T06:12:31.542 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:12:31.542 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:31.626 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:31.627 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:31.627 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:31.983 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-07-26T06:12:31.989 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-07-26T06:12:31.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-07-26T06:12:32.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-07-26T06:12:32.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-07-26T06:12:32.263 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:32.263 INFO:tasks.ceph:role: osd.2 2024-07-26T06:12:32.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:12:32.263 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:32.347 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:32.348 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:32.348 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:32.731 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-07-26T06:12:32.736 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-07-26T06:12:32.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-07-26T06:12:32.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-07-26T06:12:32.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-07-26T06:12:33.059 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:33.059 INFO:tasks.ceph:role: osd.3 2024-07-26T06:12:33.059 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:12:33.059 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:33.141 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:33.142 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:33.142 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:33.142 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:33.536 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-07-26T06:12:33.542 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-07-26T06:12:33.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-07-26T06:12:33.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-07-26T06:12:33.757 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:33.884+0000 7f06ea01a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-07-26T06:12:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:33.884+0000 7f06ea01a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-07-26T06:12:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:33.884+0000 7f06ea01a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-26T06:12:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:33.884+0000 7f06ea01a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-26T06:12:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:33.884+0000 7f06ea01a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-07-26T06:12:33.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:33.888+0000 7f06ea01a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-07-26T06:12:37.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-07-26T06:12:37.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:37.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:37.319+0000 7fb2312b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-07-26T06:12:37.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:37.320+0000 7fb2312b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-07-26T06:12:37.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:37.320+0000 7fb2312b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-26T06:12:37.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:37.320+0000 7fb2312b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-26T06:12:37.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:37.320+0000 7fb2312b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-07-26T06:12:37.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:37.322+0000 7fb2312b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-07-26T06:12:40.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-07-26T06:12:40.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:40.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:40.503+0000 7f20508b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-07-26T06:12:40.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:40.503+0000 7f20508b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-07-26T06:12:40.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:40.504+0000 7f20508b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-26T06:12:40.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:40.504+0000 7f20508b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-26T06:12:40.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:40.504+0000 7f20508b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-07-26T06:12:40.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:40.506+0000 7f20508b5640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-07-26T06:12:43.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-07-26T06:12:43.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:43.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:43.942+0000 7f79ad9a1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-07-26T06:12:43.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:43.943+0000 7f79ad9a1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-07-26T06:12:43.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:43.943+0000 7f79ad9a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-26T06:12:43.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:43.943+0000 7f79ad9a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-26T06:12:43.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:43.943+0000 7f79ad9a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-07-26T06:12:43.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:12:43.945+0000 7f79ad9a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-07-26T06:12:46.927 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-07-26T06:12:47.012 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-07-26T06:12:47.103 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:47.104 INFO:tasks.ceph:role: osd.4 2024-07-26T06:12:47.104 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:12:47.104 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-07-26T06:12:47.192 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:47.192 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:47.192 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:47.192 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:47.192 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:47.192 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:47.193 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:47.193 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:47.193 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:47.193 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:47.604 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-07-26T06:12:47.611 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-07-26T06:12:47.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-07-26T06:12:47.753 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-07-26T06:12:47.844 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-07-26T06:12:47.922 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:47.922 INFO:tasks.ceph:role: osd.5 2024-07-26T06:12:47.922 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:12:47.922 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-07-26T06:12:48.005 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:48.005 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:48.005 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:48.006 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:48.360 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-07-26T06:12:48.365 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-07-26T06:12:48.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-07-26T06:12:48.472 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-07-26T06:12:48.540 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-07-26T06:12:48.603 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:48.603 INFO:tasks.ceph:role: osd.6 2024-07-26T06:12:48.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:12:48.604 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:48.675 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:49.062 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-07-26T06:12:49.067 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-07-26T06:12:49.068 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-07-26T06:12:49.183 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-07-26T06:12:49.251 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-07-26T06:12:49.316 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-07-26T06:12:49.316 INFO:tasks.ceph:role: osd.7 2024-07-26T06:12:49.317 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:12:49.317 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-07-26T06:12:49.395 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-07-26T06:12:49.395 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-07-26T06:12:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-07-26T06:12:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-07-26T06:12:49.396 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-07-26T06:12:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-07-26T06:12:49.396 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-07-26T06:12:49.397 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-07-26T06:12:49.397 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-07-26T06:12:49.398 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-07-26T06:12:49.800 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-07-26T06:12:49.806 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-07-26T06:12:49.806 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-07-26T06:12:49.956 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-07-26T06:12:50.043 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:50.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:50.187+0000 7f04dc753640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-07-26T06:12:50.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:50.188+0000 7f04dc753640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-07-26T06:12:50.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:50.188+0000 7f04dc753640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-26T06:12:50.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:50.188+0000 7f04dc753640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-26T06:12:50.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:50.188+0000 7f04dc753640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-07-26T06:12:50.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:50.192+0000 7f04dc753640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-07-26T06:12:53.165 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-07-26T06:12:53.247 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:53.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:53.375+0000 7fd475413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-07-26T06:12:53.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:53.375+0000 7fd475413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-07-26T06:12:53.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:53.376+0000 7fd475413640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-26T06:12:53.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:53.376+0000 7fd475413640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-26T06:12:53.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:53.376+0000 7fd475413640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-07-26T06:12:53.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:53.378+0000 7fd475413640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-07-26T06:12:55.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:12:55.473 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:12:56.355 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-07-26T06:12:56.441 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:56.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:56.561+0000 7f660ecfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-07-26T06:12:56.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:56.562+0000 7f660ecfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-07-26T06:12:56.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:56.562+0000 7f660ecfd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-26T06:12:56.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:56.562+0000 7f660ecfd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-26T06:12:56.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:56.562+0000 7f660ecfd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-07-26T06:12:56.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:56.564+0000 7f660ecfd640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-07-26T06:12:59.519 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-07-26T06:12:59.595 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:12:59.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:59.720+0000 7f0b86944640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-07-26T06:12:59.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:59.721+0000 7f0b86944640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-07-26T06:12:59.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:59.721+0000 7f0b86944640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-26T06:12:59.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:59.721+0000 7f0b86944640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-26T06:12:59.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:59.721+0000 7f0b86944640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-07-26T06:12:59.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:12:59.723+0000 7f0b86944640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-07-26T06:13:02.698 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-07-26T06:13:02.785 INFO:tasks.ceph:Reading keys from all nodes... 2024-07-26T06:13:02.785 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:02.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-07-26T06:13:02.874 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:02.874 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-07-26T06:13:02.958 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:02.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-07-26T06:13:03.041 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:03.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-07-26T06:13:03.175 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:03.175 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-07-26T06:13:03.261 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:03.261 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-07-26T06:13:03.345 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:03.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-07-26T06:13:03.425 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.425 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-07-26T06:13:03.514 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.514 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-07-26T06:13:03.597 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.598 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-07-26T06:13:03.677 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.678 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-07-26T06:13:03.804 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.805 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-07-26T06:13:03.887 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.887 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-07-26T06:13:03.968 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:03.968 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-07-26T06:13:04.055 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:04.055 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-07-26T06:13:04.112 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:04.112 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-07-26T06:13:04.163 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:04.163 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-07-26T06:13:04.219 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:04.219 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-07-26T06:13:04.274 INFO:tasks.ceph:Adding keys to all mons... 2024-07-26T06:13:04.275 DEBUG:teuthology.orchestra.run.smithi163:> sudo tee -a /etc/ceph/ceph.keyring 2024-07-26T06:13:04.286 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/ceph.keyring 2024-07-26T06:13:04.361 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.y] 2024-07-26T06:13:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ== 2024-07-26T06:13:04.362 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.a] 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA== 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.c] 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw== 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.0] 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA== 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.1] 2024-07-26T06:13:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA== 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.2] 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA== 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.3] 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg== 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.x] 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ== 2024-07-26T06:13:04.364 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.b] 2024-07-26T06:13:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg== 2024-07-26T06:13:04.365 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.d] 2024-07-26T06:13:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg== 2024-07-26T06:13:04.365 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.4] 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA== 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.5] 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g== 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.6] 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ== 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.7] 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA== 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:13:04.366 INFO:teuthology.orchestra.run.smithi163.stdout:[client.2] 2024-07-26T06:13:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw== 2024-07-26T06:13:04.367 INFO:teuthology.orchestra.run.smithi163.stdout:[client.1] 2024-07-26T06:13:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ== 2024-07-26T06:13:04.367 INFO:teuthology.orchestra.run.smithi163.stdout:[client.3] 2024-07-26T06:13:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw== 2024-07-26T06:13:04.395 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.y] 2024-07-26T06:13:04.395 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ== 2024-07-26T06:13:04.396 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-07-26T06:13:04.396 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA== 2024-07-26T06:13:04.396 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-07-26T06:13:04.396 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw== 2024-07-26T06:13:04.396 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2024-07-26T06:13:04.396 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA== 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA== 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA== 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg== 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ== 2024-07-26T06:13:04.397 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg== 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg== 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.4] 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA== 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.5] 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g== 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.6] 2024-07-26T06:13:04.398 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ== 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.7] 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA== 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout:[client.2] 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw== 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ== 2024-07-26T06:13:04.399 INFO:teuthology.orchestra.run.smithi164.stdout:[client.3] 2024-07-26T06:13:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw== 2024-07-26T06:13:04.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T06:13:04.416 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T06:13:04.577 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:04.584 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:04.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:04.723 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:04.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:04.864 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:04.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.039 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.176 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.316 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T06:13:05.457 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-07-26T06:13:05.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:05.594 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:05.730 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:05.785 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-07-26T06:13:05.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:05.925 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.064 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.203 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.336 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-07-26T06:13:06.466 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.473 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.606 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.751 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:06.888 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-07-26T06:13:07.026 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-07-26T06:13:07.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-07-26T06:13:07.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-26T06:13:07.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-07-26T06:13:07.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-07-26T06:13:07.464 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-26T06:13:07.608 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-07-26T06:13:07.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-07-26T06:13:07.752 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-07-26T06:13:07.891 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-07-26T06:13:07.960 DEBUG:teuthology.orchestra.run.smithi163:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:13:07.967 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-07-26T06:13:08.020 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-07-26T06:13:08.021 INFO:tasks.ceph.mon.a:Restarting daemon 2024-07-26T06:13:08.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-07-26T06:13:08.026 INFO:tasks.ceph.mon.a:Started 2024-07-26T06:13:08.027 INFO:tasks.ceph.mon.b:Restarting daemon 2024-07-26T06:13:08.027 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-07-26T06:13:08.032 INFO:tasks.ceph.mon.b:Started 2024-07-26T06:13:08.033 INFO:tasks.ceph.mon.c:Restarting daemon 2024-07-26T06:13:08.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-07-26T06:13:08.038 INFO:tasks.ceph.mon.c:Started 2024-07-26T06:13:08.039 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-07-26T06:13:08.039 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-07-26T06:13:08.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-07-26T06:13:08.043 INFO:tasks.ceph.mgr.y:Started 2024-07-26T06:13:08.044 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-07-26T06:13:08.044 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-07-26T06:13:08.048 INFO:tasks.ceph.mgr.x:Started 2024-07-26T06:13:08.049 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-07-26T06:13:08.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-07-26T06:13:08.053 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-07-26T06:13:08.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set client debug_client 20 2024-07-26T06:13:08.100 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-07-26T06:13:08.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set client debug_ms 1 2024-07-26T06:13:08.146 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-07-26T06:13:08.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-07-26T06:13:08.195 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-07-26T06:13:08.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-07-26T06:13:08.240 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-07-26T06:13:08.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-07-26T06:13:08.289 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-07-26T06:13:08.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-07-26T06:13:08.336 INFO:tasks.ceph:setting config [mds] debug_ms = 1 2024-07-26T06:13:08.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds debug_ms 1 2024-07-26T06:13:08.383 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-07-26T06:13:08.384 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-07-26T06:13:08.432 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-07-26T06:13:08.432 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-07-26T06:13:08.445 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-07-26T06:13:08.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-07-26T06:13:08.499 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-07-26T06:13:08.499 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-07-26T06:13:08.520 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-07-26T06:13:08.520 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-07-26T06:13:08.570 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T06:13:08.570 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-07-26T06:13:08.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.589+0000 7f618aa00640 1 -- 172.21.15.163:0/1846941334 <== mon.1 v2:172.21.15.164:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v4)=0 v4) ==== 109+0+0 (secure 0 0 0) 0x7f6180010a20 con 0x7f61940d2e60 2024-07-26T06:13:08.617 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T06:13:08.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 -- 172.21.15.163:0/1846941334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61940d2e60 msgr2=0x7f6194072e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 --2- 172.21.15.163:0/1846941334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61940d2e60 0x7f6194072e90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f618000c260 tx=0x7f618000c720 comp rx=0 tx=0).stop 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 -- 172.21.15.163:0/1846941334 shutdown_connections 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 --2- 172.21.15.163:0/1846941334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f619415d540 0x7f61942034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 --2- 172.21.15.163:0/1846941334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61940d2e60 0x7f6194072e90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 --2- 172.21.15.163:0/1846941334 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61940d0540 0x7f619406fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 -- 172.21.15.163:0/1846941334 >> 172.21.15.163:0/1846941334 conn(0x7f61940b4fe0 msgr2=0x7f61940cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 -- 172.21.15.163:0/1846941334 shutdown_connections 2024-07-26T06:13:08.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.599+0000 7f6199600640 1 -- 172.21.15.163:0/1846941334 wait complete. 2024-07-26T06:13:08.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.596+0000 7f408fe00640 1 -- 172.21.15.163:0/830662145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40900ccb00 con 0x7f409015f390 2024-07-26T06:13:08.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.597+0000 7f408fe00640 1 -- 172.21.15.163:0/830662145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40900cd0c0 con 0x7f409015f390 2024-07-26T06:13:08.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.598+0000 7f408fe00640 1 -- 172.21.15.163:0/830662145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40900ce240 con 0x7f409015f390 2024-07-26T06:13:08.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.606+0000 7f4086a00640 1 -- 172.21.15.163:0/830662145 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f408801c070 con 0x7f409015f390 2024-07-26T06:13:08.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.606+0000 7f4086a00640 1 -- 172.21.15.163:0/830662145 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f408801ec50 con 0x7f409015f390 2024-07-26T06:13:08.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.606+0000 7f4086a00640 1 -- 172.21.15.163:0/830662145 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f4088023030 con 0x7f409015f390 2024-07-26T06:13:08.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.636+0000 7f0804e00640 1 -- 172.21.15.163:0/563513898 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f07d4005100 con 0x7f0800153640 2024-07-26T06:13:08.642 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-07-26T06:13:08.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-07-26T06:13:08.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.648+0000 7f07f7400640 1 -- 172.21.15.163:0/563513898 <== mon.0 v2:172.21.15.163:3300/0 9 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7f07f8012450 con 0x7f0800153640 2024-07-26T06:13:08.655 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T06:13:08.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T06:13:08.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.649+0000 7f07f7400640 1 -- 172.21.15.163:0/563513898 <== mon.0 v2:172.21.15.163:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f07f8028430 con 0x7f0800153640 2024-07-26T06:13:08.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.650+0000 7f0ed2000640 1 -- 172.21.15.163:0/3693997886 <== mon.0 v2:172.21.15.163:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f0ec40047a0 con 0x7f0ed80cfe30 2024-07-26T06:13:08.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.651+0000 7f4086a00640 1 -- 172.21.15.163:0/830662145 <== mon.0 v2:172.21.15.163:3300/0 7 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f40880145e0 con 0x7f409015f390 2024-07-26T06:13:08.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.656+0000 7f0804e00640 1 -- 172.21.15.163:0/563513898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0800153640 msgr2=0x7f0800153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.656+0000 7f0804e00640 1 --2- 172.21.15.163:0/563513898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0800153640 0x7f0800153ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0800070750 tx=0x7f07f8004700 comp rx=0 tx=0).stop 2024-07-26T06:13:08.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 -- 172.21.15.163:0/563513898 shutdown_connections 2024-07-26T06:13:08.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 --2- 172.21.15.163:0/563513898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0800153640 0x7f0800153ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 --2- 172.21.15.163:0/563513898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f080014df30 0x7f080014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 --2- 172.21.15.163:0/563513898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e4003a20 0x7f08001faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 -- 172.21.15.163:0/563513898 >> 172.21.15.163:0/563513898 conn(0x7f08000b3060 msgr2=0x7f08000ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 -- 172.21.15.163:0/563513898 shutdown_connections 2024-07-26T06:13:08.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.661+0000 7f0804e00640 1 -- 172.21.15.163:0/563513898 wait complete. 2024-07-26T06:13:08.676 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-07-26T06:13:08.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-07-26T06:13:08.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.692+0000 7ffa35800640 1 -- 172.21.15.163:0/4292372345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280323c0 msgr2=0x7ffa28052350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.692+0000 7ffa35800640 1 --2- 172.21.15.163:0/4292372345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280323c0 0x7ffa28052350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009880 tx=0x7ffa1c02a4d0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.705 DEBUG:tasks.ceph:set 18 configs 2024-07-26T06:13:08.705 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.706 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.706 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.706 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.693+0000 7ffa35800640 1 -- 172.21.15.163:0/4292372345 shutdown_connections 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.693+0000 7ffa35800640 1 --2- 172.21.15.163:0/4292372345 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280323c0 0x7ffa28052350 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.693+0000 7ffa35800640 1 --2- 172.21.15.163:0/4292372345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280dec70 0x7ffa28006500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.693+0000 7ffa35800640 1 -- 172.21.15.163:0/4292372345 >> 172.21.15.163:0/4292372345 conn(0x7ffa28036090 msgr2=0x7ffa280364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.693+0000 7ffa35800640 1 -- 172.21.15.163:0/4292372345 shutdown_connections 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.693+0000 7ffa35800640 1 -- 172.21.15.163:0/4292372345 wait complete. 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.700+0000 7ffa35800640 1 Processor -- start 2024-07-26T06:13:08.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.700+0000 7ffa35800640 1 -- start start 2024-07-26T06:13:08.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.705+0000 7ffa35800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa28009af0 0x7ffa280d49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.706+0000 7ffa35800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 0x7ffa28039c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.706+0000 7ffa35800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280dec70 0x7ffa2804c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa35800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28052a10 con 0x7ffa280dec70 2024-07-26T06:13:08.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa2fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa28009af0 0x7ffa280d49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa2f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 0x7ffa28039c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa35800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28052890 con 0x7ffa28009af0 2024-07-26T06:13:08.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa35800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa2801da20 con 0x7ffa280323c0 2024-07-26T06:13:08.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa2f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 0x7ffa28039c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:54856/0 (socket says 172.21.15.163:54856) 2024-07-26T06:13:08.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.709+0000 7ffa2f400640 1 -- 172.21.15.163:0/4140147967 learned_addr learned my addr 172.21.15.163:0/4140147967 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:08.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.710+0000 7ffa34e00640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280dec70 0x7ffa2804c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.711+0000 7ffa2f400640 1 -- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa28009af0 msgr2=0x7ffa280d49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.711+0000 7ffa2f400640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa28009af0 0x7ffa280d49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.711+0000 7ffa2f400640 1 -- 172.21.15.163:0/4140147967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280dec70 msgr2=0x7ffa2804c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.711+0000 7ffa2f400640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280dec70 0x7ffa2804c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.711+0000 7ffa2f400640 1 -- 172.21.15.163:0/4140147967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa28037550 con 0x7ffa280323c0 2024-07-26T06:13:08.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.712+0000 7ffa34e00640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280dec70 0x7ffa2804c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:08.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.712+0000 7ffa2f400640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 0x7ffa28039c10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c000c00 tx=0x7ffa1c032b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:08.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.713+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c040070 con 0x7ffa280323c0 2024-07-26T06:13:08.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.713+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7ffa1c0038c0 con 0x7ffa280323c0 2024-07-26T06:13:08.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.713+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003cd0 con 0x7ffa280323c0 2024-07-26T06:13:08.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.721+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2804a510 con 0x7ffa280323c0 2024-07-26T06:13:08.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.721+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2804aad0 con 0x7ffa280323c0 2024-07-26T06:13:08.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.727+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa2804acd0 con 0x7ffa280323c0 2024-07-26T06:13:08.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.728+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffa1c039040 con 0x7ffa280323c0 2024-07-26T06:13:08.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.728+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffa1c032020 con 0x7ffa280323c0 2024-07-26T06:13:08.737 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.746+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7ffa1c04a060 con 0x7ffa280323c0 2024-07-26T06:13:08.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.766+0000 7f81bfe00640 1 -- 172.21.15.163:0/2652134134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b806bf40 msgr2=0x7f81b806c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.766+0000 7f81bfe00640 1 --2- 172.21.15.163:0/2652134134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b806bf40 0x7f81b806c3b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f81a4005fd0 tx=0x7f81a402a560 comp rx=0 tx=0).stop 2024-07-26T06:13:08.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.767+0000 7f81bfe00640 1 -- 172.21.15.163:0/2652134134 shutdown_connections 2024-07-26T06:13:08.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.767+0000 7f81bfe00640 1 --2- 172.21.15.163:0/2652134134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b806bf40 0x7f81b806c3b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.767+0000 7f81bfe00640 1 --2- 172.21.15.163:0/2652134134 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81b80cd090 0x7f81b8159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.767+0000 7f81bfe00640 1 -- 172.21.15.163:0/2652134134 >> 172.21.15.163:0/2652134134 conn(0x7f81b80cf190 msgr2=0x7f81b80cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.767+0000 7f81bfe00640 1 -- 172.21.15.163:0/2652134134 shutdown_connections 2024-07-26T06:13:08.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.770+0000 7f81bfe00640 1 -- 172.21.15.163:0/2652134134 wait complete. 2024-07-26T06:13:08.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.775+0000 7f81bfe00640 1 Processor -- start 2024-07-26T06:13:08.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.777+0000 7f81bfe00640 1 -- start start 2024-07-26T06:13:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.779+0000 7f81bfe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81b806bf40 0x7f81b81f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.779+0000 7f81bfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b80cd090 0x7f81b81e4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.779+0000 7f81bfe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 0x7f81b81e4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.780+0000 7f81bfe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81b815d390 con 0x7f81b80cd090 2024-07-26T06:13:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.780+0000 7f81bfe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81b815d210 con 0x7f81b8159120 2024-07-26T06:13:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.780+0000 7f81be800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 0x7f81b81e4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.780+0000 7f81be800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 0x7f81b81e4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:45762/0 (socket says 172.21.15.163:45762) 2024-07-26T06:13:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.780+0000 7f81be800640 1 -- 172.21.15.163:0/3622003630 learned_addr learned my addr 172.21.15.163:0/3622003630 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.780+0000 7f81bd400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81b806bf40 0x7f81b81f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.781+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f81b815d510 con 0x7f81b806bf40 2024-07-26T06:13:08.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.781+0000 7f81be800640 1 -- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81b806bf40 msgr2=0x7f81b81f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.781+0000 7f81be800640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81b806bf40 0x7f81b81f8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.782+0000 7f81be800640 1 -- 172.21.15.163:0/3622003630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b80cd090 msgr2=0x7f81b81e4910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T06:13:08.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.785+0000 7f81bca00640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b80cd090 0x7f81b81e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.785+0000 7f81be800640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b80cd090 0x7f81b81e4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.785+0000 7f81be800640 1 -- 172.21.15.163:0/3622003630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b8074440 con 0x7f81b8159120 2024-07-26T06:13:08.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.786+0000 7f81be800640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 0x7f81b81e4e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f81a4006010 tx=0x7f81a4003890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:08.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.787+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a403f070 con 0x7f81b8159120 2024-07-26T06:13:08.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.787+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b81e2e70 con 0x7f81b8159120 2024-07-26T06:13:08.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.788+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b81e3430 con 0x7f81b8159120 2024-07-26T06:13:08.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.788+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f81a40047a0 con 0x7f81b8159120 2024-07-26T06:13:08.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.788+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4035030 con 0x7f81b8159120 2024-07-26T06:13:08.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.794+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f81a4042070 con 0x7f81b8159120 2024-07-26T06:13:08.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.794+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f81a403ba50 con 0x7f81b8159120 2024-07-26T06:13:08.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.797+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b80b28f0 con 0x7f81b8159120 2024-07-26T06:13:08.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.802+0000 7f0ee0800640 1 -- 172.21.15.163:0/3693997886 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f0eac005310 con 0x7f0ed80cfe30 2024-07-26T06:13:08.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.808+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f81b80b28f0 con 0x7f81b8159120 2024-07-26T06:13:08.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.815+0000 7f1c41a00640 1 -- 172.21.15.163:0/2549610925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 msgr2=0x7f1c34036400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.815+0000 7f1c3aa00640 1 -- 172.21.15.163:0/2549610925 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28031b90 con 0x7f1c34036020 2024-07-26T06:13:08.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.816+0000 7f1c41a00640 1 --2- 172.21.15.163:0/2549610925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c34036400 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400ea50 tx=0x7f1c2802a490 comp rx=0 tx=0).stop 2024-07-26T06:13:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.818+0000 7f0ed2000640 1 -- 172.21.15.163:0/3693997886 <== mon.0 v2:172.21.15.163:3300/0 9 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7f0ec40049f0 con 0x7f0ed80cfe30 2024-07-26T06:13:08.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.820+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa1c003e70 con 0x7ffa280323c0 2024-07-26T06:13:08.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.822+0000 7f4086a00640 1 -- 172.21.15.163:0/830662145 <== mon.0 v2:172.21.15.163:3300/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40880148c0 con 0x7f409015f390 2024-07-26T06:13:08.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.822+0000 7f0ed2000640 1 -- 172.21.15.163:0/3693997886 <== mon.0 v2:172.21.15.163:3300/0 10 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ec4004cd0 con 0x7f0ed80cfe30 2024-07-26T06:13:08.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.825+0000 7f408fe00640 1 -- 172.21.15.163:0/830662145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f40640052f0 con 0x7f409015f390 2024-07-26T06:13:08.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.826+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81a403bca0 con 0x7f81b8159120 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.823+0000 7f1c41a00640 1 -- 172.21.15.163:0/2549610925 shutdown_connections 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.823+0000 7f1c41a00640 1 --2- 172.21.15.163:0/2549610925 >> v2:172.21.15.163:3300/0 conn(0x7f1c34033590 0x7f1c3404eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.823+0000 7f1c41a00640 1 --2- 172.21.15.163:0/2549610925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c34036400 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.823+0000 7f1c41a00640 1 -- 172.21.15.163:0/2549610925 >> 172.21.15.163:0/2549610925 conn(0x7f1c3401d930 msgr2=0x7f1c340545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.825+0000 7f1c41a00640 1 -- 172.21.15.163:0/2549610925 shutdown_connections 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.825+0000 7f1c41a00640 1 -- 172.21.15.163:0/2549610925 wait complete. 2024-07-26T06:13:08.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.827+0000 7f1c41a00640 1 Processor -- start 2024-07-26T06:13:08.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.830+0000 7f1c41a00640 1 -- start start 2024-07-26T06:13:08.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.831+0000 7f1c41a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c34033590 0x7f1c340e49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.832+0000 7f1c41a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 0x7f1c3404b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.832+0000 7f1c41a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c340de9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.832+0000 7f1c41a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34054ec0 con 0x7f1c34033590 2024-07-26T06:13:08.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.832+0000 7f1c41a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34039fe0 con 0x7f1c34036020 2024-07-26T06:13:08.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.832+0000 7f1c41a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3404f0a0 con 0x7f1c34033ae0 2024-07-26T06:13:08.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.834+0000 7f1c41000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c340de9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.835+0000 7f1c3b400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 0x7f1c3404b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.838+0000 7f1c3b400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 0x7f1c3404b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:54890/0 (socket says 172.21.15.163:54890) 2024-07-26T06:13:08.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.838+0000 7f1c3b400640 1 -- 172.21.15.163:0/311263059 learned_addr learned my addr 172.21.15.163:0/311263059 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:08.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.840+0000 7f4086a00640 1 -- 172.21.15.163:0/830662145 <== mon.0 v2:172.21.15.163:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v7)=0 v7) ==== 108+0+0 (secure 0 0 0) 0x7f40900cc880 con 0x7f409015f390 2024-07-26T06:13:08.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f407be00640 1 -- 172.21.15.163:0/830662145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f409015f390 msgr2=0x7f40900c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f407be00640 1 --2- 172.21.15.163:0/830662145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f409015f390 0x7f40900c5260 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f408800b900 tx=0x7f408800bdc0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.841+0000 7f1c3be00640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c34033590 0x7f1c340e49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f407be00640 1 -- 172.21.15.163:0/830662145 shutdown_connections 2024-07-26T06:13:08.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f407be00640 1 --2- 172.21.15.163:0/830662145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f409015f390 0x7f40900c5260 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f1c3b400640 1 -- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 msgr2=0x7f1c340de9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f1c3b400640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c340de9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f1c3b400640 1 -- 172.21.15.163:0/311263059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c34033590 msgr2=0x7f1c340e49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f1c3b400640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c34033590 0x7f1c340e49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f1c3b400640 1 -- 172.21.15.163:0/311263059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c340def20 con 0x7f1c34033ae0 2024-07-26T06:13:08.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.844+0000 7f1c3be00640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c34033590 0x7f1c340e49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:08.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.842+0000 7f407be00640 1 --2- 172.21.15.163:0/830662145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40900d07e0 0x7f409015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.846+0000 7f407be00640 1 --2- 172.21.15.163:0/830662145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40900b5400 0x7f40900cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.846+0000 7f407be00640 1 -- 172.21.15.163:0/830662145 >> 172.21.15.163:0/830662145 conn(0x7f40900cfcf0 msgr2=0x7f40900d3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.844+0000 7f1c41000640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c340de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.846+0000 7f1c3b400640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 0x7f1c3404b020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000c490 tx=0x7f1c3000c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.846+0000 7f407be00640 1 -- 172.21.15.163:0/830662145 shutdown_connections 2024-07-26T06:13:08.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.848+0000 7f407be00640 1 -- 172.21.15.163:0/830662145 wait complete. 2024-07-26T06:13:08.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.848+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c30016020 con 0x7f1c34033ae0 2024-07-26T06:13:08.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.850+0000 7f0ee0800640 1 -- 172.21.15.163:0/3693997886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ed80cfe30 msgr2=0x7f0ed8157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.850+0000 7f0ee0800640 1 --2- 172.21.15.163:0/3693997886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ed80cfe30 0x7f0ed8157d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400ed30 tx=0x7f0ec400c6a0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.851+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c30012940 con 0x7f1c34033ae0 2024-07-26T06:13:08.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.851+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c30012d90 con 0x7f1c34033ae0 2024-07-26T06:13:08.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.851+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c340e4fb0 con 0x7f1c34033ae0 2024-07-26T06:13:08.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.851+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c340e3190 con 0x7f1c34033ae0 2024-07-26T06:13:08.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.855+0000 7f0ee0800640 1 -- 172.21.15.163:0/3693997886 shutdown_connections 2024-07-26T06:13:08.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.855+0000 7f0ee0800640 1 --2- 172.21.15.163:0/3693997886 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ed815d080 0x7f0ed815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.855+0000 7f0ee0800640 1 --2- 172.21.15.163:0/3693997886 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed80d2920 0x7f0ed815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.855+0000 7f0ee0800640 1 --2- 172.21.15.163:0/3693997886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ed80cfe30 0x7f0ed8157d50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.855+0000 7f0ee0800640 1 -- 172.21.15.163:0/3693997886 >> 172.21.15.163:0/3693997886 conn(0x7f0ed80b4fe0 msgr2=0x7f0ed80b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.860+0000 7f0ee0800640 1 -- 172.21.15.163:0/3693997886 shutdown_connections 2024-07-26T06:13:08.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.860+0000 7f0ee0800640 1 -- 172.21.15.163:0/3693997886 wait complete. 2024-07-26T06:13:08.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.868+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1c30019020 con 0x7f1c34033ae0 2024-07-26T06:13:08.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.868+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1c3001a890 con 0x7f1c34033ae0 2024-07-26T06:13:08.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.868+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c34031a30 con 0x7f1c34033ae0 2024-07-26T06:13:08.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.885+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f1c30020030 con 0x7f1c34033ae0 2024-07-26T06:13:08.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.924+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7ffa28052350 con 0x7ffa280323c0 2024-07-26T06:13:08.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.923+0000 7f3882200640 1 -- 172.21.15.163:0/1568755574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387400a4e0 msgr2=0x7f3874036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.923+0000 7f3882200640 1 --2- 172.21.15.163:0/1568755574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387400a4e0 0x7f3874036010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f387400ea40 tx=0x7f387002b270 comp rx=0 tx=0).stop 2024-07-26T06:13:08.930 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.930 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.927+0000 7f3882200640 1 -- 172.21.15.163:0/1568755574 shutdown_connections 2024-07-26T06:13:08.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.927+0000 7f3882200640 1 --2- 172.21.15.163:0/1568755574 >> v2:172.21.15.163:3300/0 conn(0x7f387404ff40 0x7f3874036550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.927+0000 7f3882200640 1 --2- 172.21.15.163:0/1568755574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387400a4e0 0x7f3874036010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.927+0000 7f3882200640 1 -- 172.21.15.163:0/1568755574 >> 172.21.15.163:0/1568755574 conn(0x7f387401d920 msgr2=0x7f3874054590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.930+0000 7f3882200640 1 -- 172.21.15.163:0/1568755574 shutdown_connections 2024-07-26T06:13:08.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.930+0000 7f3882200640 1 -- 172.21.15.163:0/1568755574 wait complete. 2024-07-26T06:13:08.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.932+0000 7f3882200640 1 Processor -- start 2024-07-26T06:13:08.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.936+0000 7f3882200640 1 -- start start 2024-07-26T06:13:08.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.937+0000 7ffa23e00640 1 -- 172.21.15.163:0/4140147967 <== mon.2 v2:172.21.15.164:3301/0 8 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7ffa1c032430 con 0x7ffa280323c0 2024-07-26T06:13:08.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.939+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 msgr2=0x7ffa28039c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.939+0000 7ffa35800640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 0x7ffa28039c10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c000c00 tx=0x7ffa1c032b90 comp rx=0 tx=0).stop 2024-07-26T06:13:08.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.940+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 shutdown_connections 2024-07-26T06:13:08.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.940+0000 7ffa35800640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa280dec70 0x7ffa2804c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.940+0000 7ffa35800640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa280323c0 0x7ffa28039c10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.940+0000 7ffa35800640 1 --2- 172.21.15.163:0/4140147967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa28009af0 0x7ffa280d49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.940+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 >> 172.21.15.163:0/4140147967 conn(0x7ffa28036090 msgr2=0x7ffa2804ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:08.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.941+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 shutdown_connections 2024-07-26T06:13:08.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.941+0000 7ffa35800640 1 -- 172.21.15.163:0/4140147967 wait complete. 2024-07-26T06:13:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.945+0000 7f3882200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387404ff40 0x7f38740d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.945+0000 7f3882200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 0x7f387404b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.945+0000 7f3882200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 0x7f38740d7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.949+0000 7f387aa00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 0x7f387404b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.949+0000 7f387aa00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 0x7f387404b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:54892/0 (socket says 172.21.15.163:54892) 2024-07-26T06:13:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.949+0000 7f387be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 0x7f38740d7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.950+0000 7f387be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 0x7f38740d7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45306/0 (socket says 172.21.15.163:45306) 2024-07-26T06:13:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.950+0000 7f387aa00640 1 -- 172.21.15.163:0/1344036461 learned_addr learned my addr 172.21.15.163:0/1344036461 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.950+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f38740393e0 con 0x7f387404bb10 2024-07-26T06:13:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.950+0000 7f387b400640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387404ff40 0x7f38740d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.950+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3874054eb0 con 0x7f387404ff40 2024-07-26T06:13:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.954+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3874007060 con 0x7f38740551a0 2024-07-26T06:13:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.959+0000 7f387be00640 1 -- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 msgr2=0x7f387404b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.959+0000 7f387be00640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 0x7f387404b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.959+0000 7f387be00640 1 -- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387404ff40 msgr2=0x7f38740d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.959+0000 7f387be00640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387404ff40 0x7f38740d63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.961+0000 7f387be00640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38740d6ca0 con 0x7f387404bb10 2024-07-26T06:13:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.963+0000 7f387aa00640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 0x7f387404b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.963+0000 7f387be00640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 0x7f38740d7710 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f387c06ad90 tx=0x7f387c06a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.965+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c069170 con 0x7f387404bb10 2024-07-26T06:13:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.966+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f387c08f030 con 0x7f387404bb10 2024-07-26T06:13:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.966+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387404c020 con 0x7f387404bb10 2024-07-26T06:13:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.966+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38740d71f0 con 0x7f387404bb10 2024-07-26T06:13:08.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.974+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0744b0 con 0x7f387404bb10 2024-07-26T06:13:08.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.974+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f387c0947b0 con 0x7f387404bb10 2024-07-26T06:13:08.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.974+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f387c093730 con 0x7f387404bb10 2024-07-26T06:13:08.976 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '1'], check_status=True, hostname='smithi163') 2024-07-26T06:13:08.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.977+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3858005290 con 0x7f387404bb10 2024-07-26T06:13:08.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:08.985+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f387c074650 con 0x7f387404bb10 2024-07-26T06:13:09.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.030+0000 7f5711000640 1 -- 172.21.15.163:0/3809349170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0b3250 msgr2=0x7f570c174ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.030+0000 7f5711000640 1 --2- 172.21.15.163:0/3809349170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0b3250 0x7f570c174ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5704009ca0 tx=0x7f570402bb50 comp rx=0 tx=0).stop 2024-07-26T06:13:09.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f9a75c00640 1 -- 172.21.15.163:0/3980597863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a68009af0 msgr2=0x7f9a6804eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f5711000640 1 -- 172.21.15.163:0/3809349170 shutdown_connections 2024-07-26T06:13:09.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f5711000640 1 --2- 172.21.15.163:0/3809349170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0b3250 0x7f570c174ba0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f5711000640 1 -- 172.21.15.163:0/3809349170 >> 172.21.15.163:0/3809349170 conn(0x7f570c06d280 msgr2=0x7f570c06d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f5711000640 1 -- 172.21.15.163:0/3809349170 shutdown_connections 2024-07-26T06:13:09.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f67f7e00640 1 -- 172.21.15.163:0/3151482122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f81449c0 msgr2=0x7f67f8074180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f67f7e00640 1 --2- 172.21.15.163:0/3151482122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f81449c0 0x7f67f8074180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f67e8009880 tx=0x7f67e802a620 comp rx=0 tx=0).stop 2024-07-26T06:13:09.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f5711000640 1 -- 172.21.15.163:0/3809349170 wait complete. 2024-07-26T06:13:09.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f9a75c00640 1 --2- 172.21.15.163:0/3980597863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a68009af0 0x7f9a6804eb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a7006b2e0 tx=0x7f9a700ad630 comp rx=0 tx=0).stop 2024-07-26T06:13:09.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f67f7e00640 1 -- 172.21.15.163:0/3151482122 shutdown_connections 2024-07-26T06:13:09.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f9a75c00640 1 -- 172.21.15.163:0/3980597863 shutdown_connections 2024-07-26T06:13:09.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f9a75c00640 1 --2- 172.21.15.163:0/3980597863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a68009af0 0x7f9a6804eb60 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a7006b2e0 tx=0x7f9a700ad630 comp rx=0 tx=0).stop 2024-07-26T06:13:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f9a75c00640 1 -- 172.21.15.163:0/3980597863 >> 172.21.15.163:0/3980597863 conn(0x7f9a68036090 msgr2=0x7f9a680364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f67f7e00640 1 --2- 172.21.15.163:0/3151482122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f81449c0 0x7f67f8074180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.033+0000 7f67f7e00640 1 -- 172.21.15.163:0/3151482122 >> 172.21.15.163:0/3151482122 conn(0x7f67f806d280 msgr2=0x7f67f806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.034+0000 7f67f7e00640 1 -- 172.21.15.163:0/3151482122 shutdown_connections 2024-07-26T06:13:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.037+0000 7f9a75c00640 1 -- 172.21.15.163:0/3980597863 shutdown_connections 2024-07-26T06:13:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.034+0000 7f67f7e00640 1 -- 172.21.15.163:0/3151482122 wait complete. 2024-07-26T06:13:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.037+0000 7f67f7e00640 1 Processor -- start 2024-07-26T06:13:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.037+0000 7f67f7e00640 1 -- start start 2024-07-26T06:13:09.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.039+0000 7f5711000640 1 Processor -- start 2024-07-26T06:13:09.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.039+0000 7f5711000640 1 -- start start 2024-07-26T06:13:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.040+0000 7f67f7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 0x7f67f8200eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.040+0000 7f67f7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67f81449c0 0x7f67f81f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.040+0000 7f67f7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 0x7f67f813e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.040+0000 7f67f6a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 0x7f67f8200eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.041+0000 7f67f6a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 0x7f67f8200eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:54912/0 (socket says 172.21.15.163:54912) 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.041+0000 7f67f7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 0x7f67f813e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.041+0000 7f67f7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 0x7f67f813e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45322/0 (socket says 172.21.15.163:45322) 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.041+0000 7f67f6a00640 1 -- 172.21.15.163:0/1448448456 learned_addr learned my addr 172.21.15.163:0/1448448456 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.041+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67f80b41f0 con 0x7f67f815eb40 2024-07-26T06:13:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.042+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f67f80b4070 con 0x7f67f81449c0 2024-07-26T06:13:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.042+0000 7f67f6000640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67f81449c0 0x7f67f81f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.042+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f67f80736e0 con 0x7f67f80ce4a0 2024-07-26T06:13:09.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.037+0000 7f9a75c00640 1 -- 172.21.15.163:0/3980597863 wait complete. 2024-07-26T06:13:09.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.039+0000 7f9a75c00640 1 Processor -- start 2024-07-26T06:13:09.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.043+0000 7f67f7400640 1 -- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 msgr2=0x7f67f8200eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.043+0000 7f67f7400640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 0x7f67f8200eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.043+0000 7f67f7400640 1 -- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67f81449c0 msgr2=0x7f67f81f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.043+0000 7f67f7400640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67f81449c0 0x7f67f81f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.043+0000 7f67f7400640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f80abb10 con 0x7f67f815eb40 2024-07-26T06:13:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.044+0000 7f67f7400640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 0x7f67f813e190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67f000c730 tx=0x7f67f000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.045+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0003b70 con 0x7f67f815eb40 2024-07-26T06:13:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.045+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67f0003d10 con 0x7f67f815eb40 2024-07-26T06:13:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.045+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f000aab0 con 0x7f67f815eb40 2024-07-26T06:13:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.045+0000 7f5711000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f570c08b9b0 0x7f570c203310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f67f6a00640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 0x7f67f8200eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f570b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f570c08b9b0 0x7f570c203310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f570b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f570c08b9b0 0x7f570c203310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45334/0 (socket says 172.21.15.163:45334) 2024-07-26T06:13:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f570b400640 1 -- 172.21.15.163:0/3569107407 learned_addr learned my addr 172.21.15.163:0/3569107407 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f5711000640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f570c0b3250 0x7f570c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f81403e0 con 0x7f67f815eb40 2024-07-26T06:13:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f81409a0 con 0x7f67f815eb40 2024-07-26T06:13:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.043+0000 7f9a75c00640 1 -- start start 2024-07-26T06:13:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.046+0000 7f5711000640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0cd430 0x7f570c15e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.047+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f67f000ac50 con 0x7f67f815eb40 2024-07-26T06:13:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.047+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f67f0012980 con 0x7f67f815eb40 2024-07-26T06:13:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.048+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f815f2c0 con 0x7f67f815eb40 2024-07-26T06:13:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.047+0000 7f570aa00640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f570c0b3250 0x7f570c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.047+0000 7f570be00640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0cd430 0x7f570c15e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.051+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f570c175260 con 0x7f570c08b9b0 2024-07-26T06:13:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.051+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1750e0 con 0x7f570c0b3250 2024-07-26T06:13:09.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.051+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f570c1753e0 con 0x7f570c0cd430 2024-07-26T06:13:09.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f9a75c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 0x7f9a6804d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f570aa00640 1 -- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0cd430 msgr2=0x7f570c15e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f9a75c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680e0960 0x7f9a680dc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f570aa00640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0cd430 0x7f570c15e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f570aa00640 1 -- 172.21.15.163:0/3569107407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f570c08b9b0 msgr2=0x7f570c203310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f570aa00640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f570c08b9b0 0x7f570c203310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f570aa00640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c206190 con 0x7f570c0b3250 2024-07-26T06:13:09.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.052+0000 7f570be00640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0cd430 0x7f570c15e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:09.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.053+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f67f80abb10 con 0x7f67f815eb40 2024-07-26T06:13:09.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.053+0000 7f9a6ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 0x7f9a6804d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.053+0000 7f9a6ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 0x7f9a6804d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45350/0 (socket says 172.21.15.163:45350) 2024-07-26T06:13:09.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.053+0000 7f570aa00640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f570c0b3250 0x7f570c15d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f56f800b3e0 tx=0x7f56f800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.053+0000 7f9a6ea00640 1 -- 172.21.15.163:0/3410688292 learned_addr learned my addr 172.21.15.163:0/3410688292 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.057+0000 7f9a75c00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a68021d50 0x7f9a680dde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.057+0000 7f9a75c00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6801d890 con 0x7f9a6804f0a0 2024-07-26T06:13:09.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.057+0000 7f9a75c00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6801d710 con 0x7f9a68021d50 2024-07-26T06:13:09.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.057+0000 7f9a75c00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9a6801da10 con 0x7f9a680e0960 2024-07-26T06:13:09.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.057+0000 7f9a6e000640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680e0960 0x7f9a680dc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.058+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8015020 con 0x7f570c0b3250 2024-07-26T06:13:09.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.058+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56f8011940 con 0x7f570c0b3250 2024-07-26T06:13:09.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6f400640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a68021d50 0x7f9a680dde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6ea00640 1 -- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680e0960 msgr2=0x7f9a680dc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6ea00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680e0960 0x7f9a680dc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6ea00640 1 -- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a68021d50 msgr2=0x7f9a680dde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6ea00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a68021d50 0x7f9a680dde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6ea00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6804d710 con 0x7f9a6804f0a0 2024-07-26T06:13:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.060+0000 7f9a6f400640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a68021d50 0x7f9a680dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:09.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.058+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8011d90 con 0x7f570c0b3250 2024-07-26T06:13:09.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.058+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c206470 con 0x7f570c0b3250 2024-07-26T06:13:09.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.058+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570c206a30 con 0x7f570c0b3250 2024-07-26T06:13:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.061+0000 7f9a6ea00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 0x7f9a6804d0a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00c430 tx=0x7f9a5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.061+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c017070 con 0x7f9a6804f0a0 2024-07-26T06:13:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.062+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a5c0035f0 con 0x7f9a6804f0a0 2024-07-26T06:13:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.062+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c003a20 con 0x7f9a6804f0a0 2024-07-26T06:13:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.062+0000 7f9a75c00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6804d9f0 con 0x7f9a6804f0a0 2024-07-26T06:13:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.062+0000 7f9a75c00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6804dfb0 con 0x7f9a6804f0a0 2024-07-26T06:13:09.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.067+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a44005290 con 0x7f9a6804f0a0 2024-07-26T06:13:09.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.068+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e4005290 con 0x7f570c0b3250 2024-07-26T06:13:09.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.071+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9a5c01a070 con 0x7f9a6804f0a0 2024-07-26T06:13:09.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.071+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9a5c012740 con 0x7f9a6804f0a0 2024-07-26T06:13:09.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.072+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f56f8018020 con 0x7f570c0b3250 2024-07-26T06:13:09.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.072+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f56f8019990 con 0x7f570c0b3250 2024-07-26T06:13:09.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.079+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f9a5c021070 con 0x7f9a6804f0a0 2024-07-26T06:13:09.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.079+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f56f8019d90 con 0x7f570c0b3250 2024-07-26T06:13:09.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.089+0000 7f1140c00640 1 -- 172.21.15.163:0/3645891080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11340e0950 msgr2=0x7f1134036170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.089+0000 7f1140c00640 1 --2- 172.21.15.163:0/3645891080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11340e0950 0x7f1134036170 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f112c009830 tx=0x7f112c02a590 comp rx=0 tx=0).stop 2024-07-26T06:13:09.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.090+0000 7f1140c00640 1 -- 172.21.15.163:0/3645891080 shutdown_connections 2024-07-26T06:13:09.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.090+0000 7f1140c00640 1 --2- 172.21.15.163:0/3645891080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f1134054430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.090+0000 7f1140c00640 1 --2- 172.21.15.163:0/3645891080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11340e0950 0x7f1134036170 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.090+0000 7f1140c00640 1 -- 172.21.15.163:0/3645891080 >> 172.21.15.163:0/3645891080 conn(0x7f113401fb30 msgr2=0x7f1134056860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.094+0000 7f1140c00640 1 -- 172.21.15.163:0/3645891080 shutdown_connections 2024-07-26T06:13:09.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.094+0000 7f1140c00640 1 -- 172.21.15.163:0/3645891080 wait complete. 2024-07-26T06:13:09.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.104+0000 7f0768200640 1 -- 172.21.15.163:0/3593376419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07600b2340 msgr2=0x7f0760159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.104+0000 7f0768200640 1 --2- 172.21.15.163:0/3593376419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07600b2340 0x7f0760159700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f075800b3b0 tx=0x7f075802aa90 comp rx=0 tx=0).stop 2024-07-26T06:13:09.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.101+0000 7f1140c00640 1 Processor -- start 2024-07-26T06:13:09.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.105+0000 7f0768200640 1 -- 172.21.15.163:0/3593376419 shutdown_connections 2024-07-26T06:13:09.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.105+0000 7f0768200640 1 --2- 172.21.15.163:0/3593376419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07600b2340 0x7f0760159700 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.105+0000 7f0768200640 1 -- 172.21.15.163:0/3593376419 >> 172.21.15.163:0/3593376419 conn(0x7f07600cf190 msgr2=0x7f07600cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.105+0000 7f1140c00640 1 -- start start 2024-07-26T06:13:09.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.107+0000 7f1140c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f113404a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.107+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f81b80b34d0 con 0x7f81b8159120 2024-07-26T06:13:09.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.108+0000 7f0768200640 1 -- 172.21.15.163:0/3593376419 shutdown_connections 2024-07-26T06:13:09.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.108+0000 7f1140c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1134054970 0x7f11340d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.108+0000 7f1139e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f113404a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.108+0000 7f1139e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f113404a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:54954/0 (socket says 172.21.15.163:54954) 2024-07-26T06:13:09.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.108+0000 7f0768200640 1 -- 172.21.15.163:0/3593376419 wait complete. 2024-07-26T06:13:09.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.109+0000 7f1139e00640 1 -- 172.21.15.163:0/4036709034 learned_addr learned my addr 172.21.15.163:0/4036709034 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.109+0000 7f1140c00640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1134033bc0 0x7f11340d54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.110+0000 7f1139400640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1134054970 0x7f11340d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.111+0000 7f113a800640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1134033bc0 0x7f11340d54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.112+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11340e0eb0 con 0x7f1134033bc0 2024-07-26T06:13:09.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.112+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f11340e0d30 con 0x7f1134054970 2024-07-26T06:13:09.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.112+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f11340e1030 con 0x7f113403bf20 2024-07-26T06:13:09.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.115+0000 7f1139400640 1 -- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 msgr2=0x7f113404a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.115+0000 7f1139400640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f113404a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.115+0000 7f1139400640 1 -- 172.21.15.163:0/4036709034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1134033bc0 msgr2=0x7f11340d54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.115+0000 7f1139e00640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f113404a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:09.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.116+0000 7f1139400640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1134033bc0 0x7f11340d54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.116+0000 7f1139400640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11340205a0 con 0x7f1134054970 2024-07-26T06:13:09.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.119+0000 7f0768200640 1 Processor -- start 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.116+0000 7f113a800640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1134033bc0 0x7f11340d54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.116+0000 7f1139400640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1134054970 0x7f11340d6de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f112800e9f0 tx=0x7f112800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.118+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112800ccc0 con 0x7f1134054970 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.118+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113404ab90 con 0x7f1134054970 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.119+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113404b150 con 0x7f1134054970 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.119+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1128010430 con 0x7f1134054970 2024-07-26T06:13:09.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.119+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112800ccc0 con 0x7f1134054970 2024-07-26T06:13:09.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.120+0000 7f0768200640 1 -- start start 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.121+0000 7f0768200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07600b2340 0x7f07601e2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.121+0000 7f0768200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07600b4230 0x7f07601e1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.121+0000 7f0768200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 0x7f07601e2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.122+0000 7f0768200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07600cd4a0 con 0x7f0760159320 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.122+0000 7f0768200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07600cd320 con 0x7f07600b4230 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.122+0000 7f0768200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f076006def0 con 0x7f07600b2340 2024-07-26T06:13:09.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.120+0000 7fab01000640 1 -- 172.21.15.163:0/2875081234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc15d650 msgr2=0x7faafc0d1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.120+0000 7fab01000640 1 --2- 172.21.15.163:0/2875081234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc15d650 0x7faafc0d1990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faafc0b6070 tx=0x7faaec02a620 comp rx=0 tx=0).stop 2024-07-26T06:13:09.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.123+0000 7f0766c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 0x7f07601e2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7f0766c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 0x7f07601e2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45396/0 (socket says 172.21.15.163:45396) 2024-07-26T06:13:09.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7f0766c00640 1 -- 172.21.15.163:0/3580132247 learned_addr learned my addr 172.21.15.163:0/3580132247 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7f0765800640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07600b2340 0x7f07601e2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7fab01000640 1 -- 172.21.15.163:0/2875081234 shutdown_connections 2024-07-26T06:13:09.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7f0764e00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07600b4230 0x7f07601e1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7fab01000640 1 --2- 172.21.15.163:0/2875081234 >> v2:172.21.15.163:3300/0 conn(0x7faafc06c270 0x7faafc06c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1128010680 con 0x7f1134054970 2024-07-26T06:13:09.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7fab01000640 1 --2- 172.21.15.163:0/2875081234 >> v2:172.21.15.164:3300/0 conn(0x7faafc0b53b0 0x7faafc0b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.124+0000 7fab01000640 1 --2- 172.21.15.163:0/2875081234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc15d650 0x7faafc0d1990 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.125+0000 7fab01000640 1 -- 172.21.15.163:0/2875081234 >> 172.21.15.163:0/2875081234 conn(0x7faafc0d36b0 msgr2=0x7faafc0d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.125+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1114005290 con 0x7f1134054970 2024-07-26T06:13:09.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.126+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1128018970 con 0x7f1134054970 2024-07-26T06:13:09.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.126+0000 7f81b5600640 1 -- 172.21.15.163:0/3622003630 <== mon.1 v2:172.21.15.164:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7f81b80b34d0 con 0x7f81b8159120 2024-07-26T06:13:09.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.127+0000 7f0766c00640 1 -- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07600b2340 msgr2=0x7f07601e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.127+0000 7f0766c00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07600b2340 0x7f07601e2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.127+0000 7f0766c00640 1 -- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07600b4230 msgr2=0x7f07601e1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.127+0000 7f0766c00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07600b4230 0x7f07601e1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 msgr2=0x7f81b81e4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 0x7f81b81e4e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f81a4006010 tx=0x7f81a4003890 comp rx=0 tx=0).stop 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 shutdown_connections 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81b8159120 0x7f81b81e4e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81b80cd090 0x7f81b81e4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 --2- 172.21.15.163:0/3622003630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81b806bf40 0x7f81b81f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 >> 172.21.15.163:0/3622003630 conn(0x7f81b80cf190 msgr2=0x7f81b815b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 shutdown_connections 2024-07-26T06:13:09.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.128+0000 7f81bfe00640 1 -- 172.21.15.163:0/3622003630 wait complete. 2024-07-26T06:13:09.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.129+0000 7fab01000640 1 -- 172.21.15.163:0/2875081234 shutdown_connections 2024-07-26T06:13:09.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.129+0000 7fab01000640 1 -- 172.21.15.163:0/2875081234 wait complete. 2024-07-26T06:13:09.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.130+0000 7f0766c00640 1 -- 172.21.15.163:0/3580132247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07600c4600 con 0x7f0760159320 2024-07-26T06:13:09.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.131+0000 7fab01000640 1 Processor -- start 2024-07-26T06:13:09.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.132+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f1128018420 con 0x7f1134054970 2024-07-26T06:13:09.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.134+0000 7fab01000640 1 -- start start 2024-07-26T06:13:09.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.134+0000 7f0765800640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07600b2340 0x7f07601e2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:09.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.134+0000 7f0764e00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07600b4230 0x7f07601e1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:09.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.134+0000 7f0766c00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 0x7f07601e2040 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f074c00c3f0 tx=0x7f074c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.135+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c011790 con 0x7f0760159320 2024-07-26T06:13:09.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.135+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f074c017070 con 0x7f0760159320 2024-07-26T06:13:09.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.135+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c011cc0 con 0x7f0760159320 2024-07-26T06:13:09.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.136+0000 7f0768200640 1 -- 172.21.15.163:0/3580132247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076013fab0 con 0x7f0760159320 2024-07-26T06:13:09.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.139+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f1c34031750 con 0x7f1c34033ae0 2024-07-26T06:13:09.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.139+0000 7f0768200640 1 -- 172.21.15.163:0/3580132247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07600c48e0 con 0x7f0760159320 2024-07-26T06:13:09.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.140+0000 7fab01000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 0x7faafc06fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.140+0000 7f563fe00640 1 -- 172.21.15.163:0/3970421982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 msgr2=0x7f56400b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.141+0000 7f563fe00640 1 --2- 172.21.15.163:0/3970421982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400b64c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f56400d4420 tx=0x7f562802a460 comp rx=0 tx=0).stop 2024-07-26T06:13:09.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7f563fe00640 1 -- 172.21.15.163:0/3970421982 shutdown_connections 2024-07-26T06:13:09.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7f563fe00640 1 --2- 172.21.15.163:0/3970421982 >> v2:172.21.15.163:3300/0 conn(0x7f56400b1e80 0x7f56400b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7f563fe00640 1 --2- 172.21.15.163:0/3970421982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400b64c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7f563fe00640 1 --2- 172.21.15.163:0/3970421982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f564015d5a0 0x7f56400ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7faaf9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 0x7faafc06fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7faaf9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 0x7faafc06fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45422/0 (socket says 172.21.15.163:45422) 2024-07-26T06:13:09.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7faaf9e00640 1 -- 172.21.15.163:0/1824232923 learned_addr learned my addr 172.21.15.163:0/1824232923 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7f563fe00640 1 -- 172.21.15.163:0/3970421982 >> 172.21.15.163:0/3970421982 conn(0x7f56400cfda0 msgr2=0x7f56400d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7fab01000640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc0b53b0 0x7faafc1fa800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.143+0000 7fab01000640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faafc15d650 0x7faafc0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.144+0000 7faaf9400640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc0b53b0 0x7faafc1fa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.145+0000 7f563fe00640 1 -- 172.21.15.163:0/3970421982 shutdown_connections 2024-07-26T06:13:09.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.141+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0734005290 con 0x7f0760159320 2024-07-26T06:13:09.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.144+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f074c01c460 con 0x7f0760159320 2024-07-26T06:13:09.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.145+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f074c012780 con 0x7f0760159320 2024-07-26T06:13:09.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.145+0000 7faafa800640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faafc15d650 0x7faafc0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.145+0000 7f563fe00640 1 -- 172.21.15.163:0/3970421982 wait complete. 2024-07-26T06:13:09.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.146+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faafc0d2050 con 0x7faafc06c270 2024-07-26T06:13:09.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.146+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faafc0d1ed0 con 0x7faafc15d650 2024-07-26T06:13:09.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.147+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faafc06cc40 con 0x7faafc0b53b0 2024-07-26T06:13:09.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.144+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f3858005580 con 0x7f387404bb10 2024-07-26T06:13:09.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.148+0000 7f563fe00640 1 Processor -- start 2024-07-26T06:13:09.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.148+0000 7f563fe00640 1 -- start start 2024-07-26T06:13:09.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.149+0000 7faaf9e00640 1 -- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc0b53b0 msgr2=0x7faafc1fa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.150+0000 7faaf9e00640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc0b53b0 0x7faafc1fa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.150+0000 7faaf9e00640 1 -- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faafc15d650 msgr2=0x7faafc0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.150+0000 7faaf9e00640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faafc15d650 0x7faafc0cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.150+0000 7faaf9e00640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc2015b0 con 0x7faafc06c270 2024-07-26T06:13:09.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.150+0000 7faaf9400640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc0b53b0 0x7faafc1fa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:09.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.151+0000 7faaf9e00640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 0x7faafc06fde0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faaec009850 tx=0x7faaec004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.153+0000 7f2750a00640 1 -- 172.21.15.163:0/3092282566 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c0ad8f0 con 0x7f27480e1860 2024-07-26T06:13:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.153+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec041070 con 0x7faafc06c270 2024-07-26T06:13:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.154+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaec031960 con 0x7faafc06c270 2024-07-26T06:13:09.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.152+0000 7f563fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56400b1e80 0x7f56400c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.152+0000 7f563fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.152+0000 7f563fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f564015d5a0 0x7f56400c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.152+0000 7f563fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f56400b4da0 con 0x7f564015d5a0 2024-07-26T06:13:09.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.152+0000 7f563fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f56400b2690 con 0x7f56400ce830 2024-07-26T06:13:09.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.152+0000 7f563fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f56400b4f20 con 0x7f56400b1e80 2024-07-26T06:13:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.154+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec031db0 con 0x7faafc06c270 2024-07-26T06:13:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.155+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faafc201890 con 0x7faafc06c270 2024-07-26T06:13:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.156+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faafc06f5e0 con 0x7faafc06c270 2024-07-26T06:13:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.157+0000 7f1c2fe00640 1 -- 172.21.15.163:0/311263059 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v10)=0 v10) ==== 115+0+0 (secure 0 0 0) 0x7f1c30020030 con 0x7f1c34033ae0 2024-07-26T06:13:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.158+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f074c011e60 con 0x7f0760159320 2024-07-26T06:13:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.158+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 msgr2=0x7f1c3404b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f1c41a00640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 0x7f1c3404b020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000c490 tx=0x7f1c3000c950 comp rx=0 tx=0).stop 2024-07-26T06:13:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.155+0000 7f2753200640 1 -- 172.21.15.163:0/3092282566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27480e1860 msgr2=0x7f2748056ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.155+0000 7f2753200640 1 --2- 172.21.15.163:0/3092282566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27480e1860 0x7f2748056ec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f274c07a7e0 tx=0x7f274c0a5280 comp rx=0 tx=0).stop 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 shutdown_connections 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f1c41a00640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c34036020 0x7f1c340de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f1c41a00640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34033ae0 0x7f1c3404b020 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000c490 tx=0x7f1c3000c950 comp rx=0 tx=0).stop 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f1c41a00640 1 --2- 172.21.15.163:0/311263059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c34033590 0x7f1c340e49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 >> 172.21.15.163:0/311263059 conn(0x7f1c3401d930 msgr2=0x7f1c34033fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f2753200640 1 -- 172.21.15.163:0/3092282566 shutdown_connections 2024-07-26T06:13:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f2753200640 1 --2- 172.21.15.163:0/3092282566 >> v2:172.21.15.163:3300/0 conn(0x7f2748053600 0x7f2748054380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f2753200640 1 --2- 172.21.15.163:0/3092282566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27480e1860 0x7f2748056ec0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.159+0000 7f2753200640 1 -- 172.21.15.163:0/3092282566 >> 172.21.15.163:0/3092282566 conn(0x7f274801fb40 msgr2=0x7f27480567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 shutdown_connections 2024-07-26T06:13:09.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f563f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f564015d5a0 0x7f56400c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f563f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f564015d5a0 0x7f56400c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45436/0 (socket says 172.21.15.163:45436) 2024-07-26T06:13:09.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f563f400640 1 -- 172.21.15.163:0/3691912893 learned_addr learned my addr 172.21.15.163:0/3691912893 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f5636000640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f563ea00640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56400b1e80 0x7f56400c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f1c41a00640 1 -- 172.21.15.163:0/311263059 wait complete. 2024-07-26T06:13:09.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.161+0000 7f386f400640 1 -- 172.21.15.163:0/1344036461 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v11)=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7f387c074830 con 0x7f387404bb10 2024-07-26T06:13:09.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.160+0000 7f2753200640 1 -- 172.21.15.163:0/3092282566 shutdown_connections 2024-07-26T06:13:09.164 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 msgr2=0x7f38740d7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 0x7f38740d7710 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f387c06ad90 tx=0x7f387c06a5c0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.162+0000 7f5636000640 1 -- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56400b1e80 msgr2=0x7f56400c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.162+0000 7f5636000640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56400b1e80 0x7f56400c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.162+0000 7f5636000640 1 -- 172.21.15.163:0/3691912893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f564015d5a0 msgr2=0x7f56400c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.162+0000 7f5636000640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f564015d5a0 0x7f56400c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f5636000640 1 -- 172.21.15.163:0/3691912893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56400cb7d0 con 0x7f56400ce830 2024-07-26T06:13:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faafc069e60 con 0x7faafc06c270 2024-07-26T06:13:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 shutdown_connections 2024-07-26T06:13:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f387404bb10 0x7f38740d7710 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740551a0 0x7f387404b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 --2- 172.21.15.163:0/1344036461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f387404ff40 0x7f38740d63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 >> 172.21.15.163:0/1344036461 conn(0x7f387401d920 msgr2=0x7f38740522d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.163+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 shutdown_connections 2024-07-26T06:13:09.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.164+0000 7f3882200640 1 -- 172.21.15.163:0/1344036461 wait complete. 2024-07-26T06:13:09.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.166+0000 7f5636000640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400cc660 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5628004000 tx=0x7f562802a980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.167+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628040030 con 0x7f56400ce830 2024-07-26T06:13:09.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.167+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f562803c530 con 0x7f56400ce830 2024-07-26T06:13:09.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.162+0000 7f2753200640 1 -- 172.21.15.163:0/3092282566 wait complete. 2024-07-26T06:13:09.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.167+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562803c900 con 0x7f56400ce830 2024-07-26T06:13:09.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.167+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56400cbab0 con 0x7f56400ce830 2024-07-26T06:13:09.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.167+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faaec044070 con 0x7faafc06c270 2024-07-26T06:13:09.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.168+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56400c8b50 con 0x7f56400ce830 2024-07-26T06:13:09.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.168+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5628043030 con 0x7f56400ce830 2024-07-26T06:13:09.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.168+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7faaec046450 con 0x7faafc06c270 2024-07-26T06:13:09.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.169+0000 7f2753200640 1 Processor -- start 2024-07-26T06:13:09.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.169+0000 7f2753200640 1 -- start start 2024-07-26T06:13:09.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.169+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5628031d60 con 0x7f56400ce830 2024-07-26T06:13:09.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56400ced00 con 0x7f56400ce830 2024-07-26T06:13:09.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7faafc069e60 con 0x7faafc06c270 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f2753200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2748053600 0x7f274803bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f2753200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 0x7f274804cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f2753200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27480e1860 0x7f2748185a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f2753200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2748054a40 con 0x7f2748057400 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f2753200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f27480548c0 con 0x7f2748053600 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.171+0000 7f2753200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2748054bc0 con 0x7f27480e1860 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 0x7f274804cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 0x7f274804cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45442/0 (socket says 172.21.15.163:45442) 2024-07-26T06:13:09.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 -- 172.21.15.163:0/1272227279 learned_addr learned my addr 172.21.15.163:0/1272227279 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2752800640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27480e1860 0x7f2748185a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 -- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27480e1860 msgr2=0x7f2748185a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27480e1860 0x7f2748185a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 -- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2748053600 msgr2=0x7f274803bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2748053600 0x7f274803bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.172+0000 7f2751400640 1 -- 172.21.15.163:0/1272227279 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274803a2b0 con 0x7f2748057400 2024-07-26T06:13:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.173+0000 7f2751400640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 0x7f274804cab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f274000c450 tx=0x7f274000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.175+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400052d0 con 0x7f2748057400 2024-07-26T06:13:09.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.175+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2740005470 con 0x7f2748057400 2024-07-26T06:13:09.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.175+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740012870 con 0x7f2748057400 2024-07-26T06:13:09.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.176+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274803a590 con 0x7f2748057400 2024-07-26T06:13:09.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.176+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274803a870 con 0x7f2748057400 2024-07-26T06:13:09.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.177+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2740012a10 con 0x7f2748057400 2024-07-26T06:13:09.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.177+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f274001b8b0 con 0x7f2748057400 2024-07-26T06:13:09.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.180+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f56400ced00 con 0x7f56400ce830 2024-07-26T06:13:09.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.178+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2748055070 con 0x7f2748057400 2024-07-26T06:13:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.190+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f274001bab0 con 0x7f2748057400 2024-07-26T06:13:09.216 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.217 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.237+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f56e4005850 con 0x7f570c0b3250 2024-07-26T06:13:09.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.244+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f67f815b620 con 0x7f67f815eb40 2024-07-26T06:13:09.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.248+0000 7f5703400640 1 -- 172.21.15.163:0/3569107407 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v12)=0 v12) ==== 121+0+0 (secure 0 0 0) 0x7f56f80193b0 con 0x7f570c0b3250 2024-07-26T06:13:09.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.250+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f570c0b3250 msgr2=0x7f570c15d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.250+0000 7f5711000640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f570c0b3250 0x7f570c15d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f56f800b3e0 tx=0x7f56f800b8a0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.250+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 shutdown_connections 2024-07-26T06:13:09.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.250+0000 7f5711000640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f570c0cd430 0x7f570c15e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.251+0000 7f5711000640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f570c0b3250 0x7f570c15d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.251+0000 7f5711000640 1 --2- 172.21.15.163:0/3569107407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f570c08b9b0 0x7f570c203310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.251+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 >> 172.21.15.163:0/3569107407 conn(0x7f570c06d280 msgr2=0x7f570c1692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.252+0000 7f67ef400640 1 -- 172.21.15.163:0/1448448456 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v13)=0 v13) ==== 119+0+0 (secure 0 0 0) 0x7f67f0012bd0 con 0x7f67f815eb40 2024-07-26T06:13:09.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.252+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 shutdown_connections 2024-07-26T06:13:09.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.252+0000 7f5711000640 1 -- 172.21.15.163:0/3569107407 wait complete. 2024-07-26T06:13:09.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 msgr2=0x7f67f813e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 0x7f67f813e190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67f000c730 tx=0x7f67f000cbf0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 shutdown_connections 2024-07-26T06:13:09.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67f815eb40 0x7f67f813e190 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67f81449c0 0x7f67f81f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 --2- 172.21.15.163:0/1448448456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67f80ce4a0 0x7f67f8200eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 >> 172.21.15.163:0/1448448456 conn(0x7f67f806d280 msgr2=0x7f67f815dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.259+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f9a44005850 con 0x7f9a6804f0a0 2024-07-26T06:13:09.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.260+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 shutdown_connections 2024-07-26T06:13:09.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.260+0000 7f67f7e00640 1 -- 172.21.15.163:0/1448448456 wait complete. 2024-07-26T06:13:09.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.277+0000 7f9a66a00640 1 -- 172.21.15.163:0/3410688292 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v14)=0 v14) ==== 121+0+0 (secure 0 0 0) 0x7f9a5c003bc0 con 0x7f9a6804f0a0 2024-07-26T06:13:09.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.279+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 msgr2=0x7f9a6804d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.279+0000 7f9a5be00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 0x7f9a6804d0a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00c430 tx=0x7f9a5c00c8f0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 shutdown_connections 2024-07-26T06:13:09.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a68021d50 0x7f9a680dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680e0960 0x7f9a680dc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 --2- 172.21.15.163:0/3410688292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a6804f0a0 0x7f9a6804d0a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 >> 172.21.15.163:0/3410688292 conn(0x7f9a68036090 msgr2=0x7f9a68006470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 shutdown_connections 2024-07-26T06:13:09.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.280+0000 7f9a5be00640 1 -- 172.21.15.163:0/3410688292 wait complete. 2024-07-26T06:13:09.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.301+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7faafc0752b0 con 0x7faafc06c270 2024-07-26T06:13:09.305 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.305 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.305 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.307+0000 7faaf2000640 1 -- 172.21.15.163:0/1824232923 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v15)=0 v15) ==== 120+0+0 (secure 0 0 0) 0x7faafc0752b0 con 0x7faafc06c270 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 msgr2=0x7faafc06fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 0x7faafc06fde0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faaec009850 tx=0x7faaec004260 comp rx=0 tx=0).stop 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 shutdown_connections 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faafc15d650 0x7faafc0cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faafc0b53b0 0x7faafc1fa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 --2- 172.21.15.163:0/1824232923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faafc06c270 0x7faafc06fde0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 >> 172.21.15.163:0/1824232923 conn(0x7faafc0d36b0 msgr2=0x7faafc0b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 shutdown_connections 2024-07-26T06:13:09.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.309+0000 7fab01000640 1 -- 172.21.15.163:0/1824232923 wait complete. 2024-07-26T06:13:09.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.310+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f0734005580 con 0x7f0760159320 2024-07-26T06:13:09.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.315+0000 7f075d600640 1 -- 172.21.15.163:0/3580132247 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v16)=0 v16) ==== 125+0+0 (secure 0 0 0) 0x7f074c004020 con 0x7f0760159320 2024-07-26T06:13:09.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.316+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 msgr2=0x7f07601e2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.316+0000 7f0752a00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 0x7f07601e2040 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f074c00c3f0 tx=0x7f074c00c8b0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 shutdown_connections 2024-07-26T06:13:09.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0760159320 0x7f07601e2040 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07600b4230 0x7f07601e1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 --2- 172.21.15.163:0/3580132247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07600b2340 0x7f07601e2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 >> 172.21.15.163:0/3580132247 conn(0x7f07600cf190 msgr2=0x7f0760156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 shutdown_connections 2024-07-26T06:13:09.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.317+0000 7f0752a00640 1 -- 172.21.15.163:0/3580132247 wait complete. 2024-07-26T06:13:09.327 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.332+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f1114005850 con 0x7f1134054970 2024-07-26T06:13:09.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.340+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f274800ddc0 con 0x7f2748057400 2024-07-26T06:13:09.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.340+0000 7f1132000640 1 -- 172.21.15.163:0/4036709034 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v17)=0 v17) ==== 120+0+0 (secure 0 0 0) 0x7f1128018b70 con 0x7f1134054970 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.341+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1134054970 msgr2=0x7f11340d6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1134054970 0x7f11340d6de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f112800e9f0 tx=0x7f112800edf0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 shutdown_connections 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1134033bc0 0x7f11340d54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1134054970 0x7f11340d6de0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 --2- 172.21.15.163:0/4036709034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113403bf20 0x7f113404a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 >> 172.21.15.163:0/4036709034 conn(0x7f113401fb30 msgr2=0x7f11340da170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 shutdown_connections 2024-07-26T06:13:09.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.342+0000 7f1140c00640 1 -- 172.21.15.163:0/4036709034 wait complete. 2024-07-26T06:13:09.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.344+0000 7f2746000640 1 -- 172.21.15.163:0/1272227279 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v18)=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f274001bc90 con 0x7f2748057400 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 msgr2=0x7f274804cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 0x7f274804cab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f274000c450 tx=0x7f274000c910 comp rx=0 tx=0).stop 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 shutdown_connections 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27480e1860 0x7f2748185a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2748057400 0x7f274804cab0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 --2- 172.21.15.163:0/1272227279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2748053600 0x7f274803bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 >> 172.21.15.163:0/1272227279 conn(0x7f274801fb40 msgr2=0x7f2748007ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 shutdown_connections 2024-07-26T06:13:09.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.346+0000 7f2753200640 1 -- 172.21.15.163:0/1272227279 wait complete. 2024-07-26T06:13:09.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.348+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f56400ce2f0 con 0x7f56400ce830 2024-07-26T06:13:09.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.356+0000 7f5637400640 1 -- 172.21.15.163:0/3691912893 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f5628004300 con 0x7f56400ce830 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.357+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 msgr2=0x7f56400cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.357+0000 7f563fe00640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400cc660 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5628004000 tx=0x7f562802a980 comp rx=0 tx=0).stop 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 shutdown_connections 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f564015d5a0 0x7f56400c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56400ce830 0x7f56400cc660 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 --2- 172.21.15.163:0/3691912893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56400b1e80 0x7f56400c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 >> 172.21.15.163:0/3691912893 conn(0x7f56400cfda0 msgr2=0x7f56400d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 shutdown_connections 2024-07-26T06:13:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.358+0000 7f563fe00640 1 -- 172.21.15.163:0/3691912893 wait complete. 2024-07-26T06:13:09.362 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.365 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.365 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi163') 2024-07-26T06:13:09.374 INFO:tasks.ceph:Setting crush tunables to default 2024-07-26T06:13:09.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd crush tunables default 2024-07-26T06:13:09.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 -- 172.21.15.163:0/3389147192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 msgr2=0x7f6e700b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/3389147192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e700b64c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e58005fd0 tx=0x7f6e5802a4b0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 -- 172.21.15.163:0/3389147192 shutdown_connections 2024-07-26T06:13:09.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/3389147192 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e700b64c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/3389147192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 0x7f6e700ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 -- 172.21.15.163:0/3389147192 >> 172.21.15.163:0/3389147192 conn(0x7f6e700cfda0 msgr2=0x7f6e700d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 -- 172.21.15.163:0/3389147192 shutdown_connections 2024-07-26T06:13:09.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.591+0000 7f6e6fe00640 1 -- 172.21.15.163:0/3389147192 wait complete. 2024-07-26T06:13:09.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.593+0000 7f6e6fe00640 1 Processor -- start 2024-07-26T06:13:09.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.593+0000 7f6e6fe00640 1 -- start start 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e700b1e80 0x7f6e701554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e70154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 0x7f6e7014b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e700b5130 con 0x7f6e700ce830 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e700b4fb0 con 0x7f6e700b1e80 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e700b52b0 con 0x7f6e7015d5a0 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e70154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e70154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45458/0 (socket says 172.21.15.163:45458) 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e67e00640 1 -- 172.21.15.163:0/4150462187 learned_addr learned my addr 172.21.15.163:0/4150462187 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6ea00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e700b1e80 0x7f6e701554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e67e00640 1 -- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 msgr2=0x7f6e7014b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.594+0000 7f6e6f400640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 0x7f6e7014b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e67e00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 0x7f6e7014b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e67e00640 1 -- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e700b1e80 msgr2=0x7f6e701554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e67e00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e700b1e80 0x7f6e701554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e67e00640 1 -- 172.21.15.163:0/4150462187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e701f3e60 con 0x7f6e700ce830 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e6ea00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e700b1e80 0x7f6e701554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e6f400640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 0x7f6e7014b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e67e00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e70154720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006010 tx=0x7f6e580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5803f070 con 0x7f6e700ce830 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e580047a0 con 0x7f6e700ce830 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58035030 con 0x7f6e700ce830 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e70158600 con 0x7f6e700ce830 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.595+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e700c8da0 con 0x7f6e700ce830 2024-07-26T06:13:09.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.596+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6e58042070 con 0x7f6e700ce830 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.596+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6e5803ba50 con 0x7f6e700ce830 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.596+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e700b3930 con 0x7f6e700ce830 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.598+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f6e700b3930 con 0x7f6e700ce830 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.667+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6e700ce2f0 con 0x7f6e700ce830 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.670+0000 7f6e66a00640 1 -- 172.21.15.163:0/4150462187 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6e701f3e60 con 0x7f6e700ce830 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:adjusted tunables profile to default 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 msgr2=0x7f6e70154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e70154720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006010 tx=0x7f6e580038a0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 shutdown_connections 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e7015d5a0 0x7f6e7014b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e700ce830 0x7f6e70154720 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 --2- 172.21.15.163:0/4150462187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e700b1e80 0x7f6e701554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 >> 172.21.15.163:0/4150462187 conn(0x7f6e700cfda0 msgr2=0x7f6e700d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:09.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 shutdown_connections 2024-07-26T06:13:09.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:09.672+0000 7f6e6fe00640 1 -- 172.21.15.163:0/4150462187 wait complete. 2024-07-26T06:13:09.689 INFO:tasks.ceph:check_enable_crimson: False 2024-07-26T06:13:09.689 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-07-26T06:13:09.689 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:09.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-07-26T06:13:09.762 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:09.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-07-26T06:13:09.875 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:09.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-07-26T06:13:09.990 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:09.990 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-07-26T06:13:10.107 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:10.107 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-07-26T06:13:10.182 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:10.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-07-26T06:13:10.254 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:10.254 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-07-26T06:13:10.366 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:13:10.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-07-26T06:13:10.486 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 72caccc6-2cb3-4d5d-bb11-9eacb422aeec 0 2024-07-26T06:13:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.670+0000 7f2423e00640 1 -- 172.21.15.164:0/4146327137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242415d5a0 msgr2=0x7f24240ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.670+0000 7f2423e00640 1 --2- 172.21.15.164:0/4146327137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242415d5a0 0x7f24240ce2f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f24240690b0 tx=0x7f241402a4d0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.670+0000 7f2423e00640 1 -- 172.21.15.164:0/4146327137 shutdown_connections 2024-07-26T06:13:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.670+0000 7f2423e00640 1 --2- 172.21.15.164:0/4146327137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24240b1e80 0x7f24240b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.670+0000 7f2423e00640 1 --2- 172.21.15.164:0/4146327137 >> v2:172.21.15.163:3300/0 conn(0x7f24240ce830 0x7f24240b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.670+0000 7f2423e00640 1 --2- 172.21.15.164:0/4146327137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242415d5a0 0x7f24240ce2f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.671+0000 7f2423e00640 1 -- 172.21.15.164:0/4146327137 >> 172.21.15.164:0/4146327137 conn(0x7f24240cfda0 msgr2=0x7f24240d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.671+0000 7f2423e00640 1 -- 172.21.15.164:0/4146327137 shutdown_connections 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.671+0000 7f2423e00640 1 -- 172.21.15.164:0/4146327137 wait complete. 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.673+0000 7f2423e00640 1 Processor -- start 2024-07-26T06:13:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.673+0000 7f2423e00640 1 -- start start 2024-07-26T06:13:10.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24240b1e80 0x7f24240cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:10.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24240ce830 0x7f24241f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:10.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 0x7f24240cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:10.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24240b2760 con 0x7f24240ce830 2024-07-26T06:13:10.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f24240b25e0 con 0x7f24240b1e80 2024-07-26T06:13:10.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f24240b4da0 con 0x7f242415d5a0 2024-07-26T06:13:10.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2422a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24240b1e80 0x7f24240cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:10.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 0x7f24240cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:10.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 0x7f24240cc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58642/0 (socket says 172.21.15.164:58642) 2024-07-26T06:13:10.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2423400640 1 -- 172.21.15.164:0/502623057 learned_addr learned my addr 172.21.15.164:0/502623057 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2422a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24240b1e80 0x7f24240cb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43690/0 (socket says 172.21.15.164:43690) 2024-07-26T06:13:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.674+0000 7f2422000640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24240ce830 0x7f24241f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423400640 1 -- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24240b1e80 msgr2=0x7f24240cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423400640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24240b1e80 0x7f24240cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423400640 1 -- 172.21.15.164:0/502623057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24240ce830 msgr2=0x7f24241f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423400640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24240ce830 0x7f24241f46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423400640 1 -- 172.21.15.164:0/502623057 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24240ca880 con 0x7f242415d5a0 2024-07-26T06:13:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423400640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 0x7f24240cc6b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f241c009fd0 tx=0x7f241c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c003b70 con 0x7f242415d5a0 2024-07-26T06:13:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f241c003d10 con 0x7f242415d5a0 2024-07-26T06:13:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423e00640 1 -- 172.21.15.164:0/502623057 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24240cab60 con 0x7f242415d5a0 2024-07-26T06:13:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c004280 con 0x7f242415d5a0 2024-07-26T06:13:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f2423e00640 1 -- 172.21.15.164:0/502623057 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24240cb120 con 0x7f242415d5a0 2024-07-26T06:13:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.675+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f241c021440 con 0x7f242415d5a0 2024-07-26T06:13:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.676+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f241c018860 con 0x7f242415d5a0 2024-07-26T06:13:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.676+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f8005290 con 0x7f242415d5a0 2024-07-26T06:13:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.679+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f241c038030 con 0x7f242415d5a0 2024-07-26T06:13:10.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.757+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "72caccc6-2cb3-4d5d-bb11-9eacb422aeec", "id": 0} v 0) -- 0x7f23f8005850 con 0x7f242415d5a0 2024-07-26T06:13:10.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.764+0000 7f241b400640 1 -- 172.21.15.164:0/502623057 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72caccc6-2cb3-4d5d-bb11-9eacb422aeec", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f241c026030 con 0x7f242415d5a0 2024-07-26T06:13:10.765 INFO:teuthology.orchestra.run.smithi164.stdout:0 2024-07-26T06:13:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 msgr2=0x7f24240cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 0x7f24240cc6b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f241c009fd0 tx=0x7f241c00c6a0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 shutdown_connections 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f242415d5a0 0x7f24240cc6b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24240ce830 0x7f24241f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 --2- 172.21.15.164:0/502623057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24240b1e80 0x7f24240cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 >> 172.21.15.164:0/502623057 conn(0x7f24240cfda0 msgr2=0x7f24240b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 shutdown_connections 2024-07-26T06:13:10.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.766+0000 7f2418c00640 1 -- 172.21.15.164:0/502623057 wait complete. 2024-07-26T06:13:10.784 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 69942df7-d9b8-4051-b4ec-422cf7beee9b 1 2024-07-26T06:13:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 -- 172.21.15.164:0/3538232492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232815d5a0 msgr2=0x7f23280ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 --2- 172.21.15.164:0/3538232492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232815d5a0 0x7f23280ce2f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f231c009900 tx=0x7f231c02a490 comp rx=0 tx=0).stop 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 -- 172.21.15.164:0/3538232492 shutdown_connections 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 --2- 172.21.15.164:0/3538232492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232815d5a0 0x7f23280ce2f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 -- 172.21.15.164:0/3538232492 >> 172.21.15.164:0/3538232492 conn(0x7f23280cfda0 msgr2=0x7f23280d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 -- 172.21.15.164:0/3538232492 shutdown_connections 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.965+0000 7f232e400640 1 -- 172.21.15.164:0/3538232492 wait complete. 2024-07-26T06:13:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.967+0000 7f232e400640 1 Processor -- start 2024-07-26T06:13:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.967+0000 7f232e400640 1 -- start start 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232e400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 0x7f23280c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232e400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23280ce830 0x7f232815f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232e400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232815d5a0 0x7f232815f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232e400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f23280b6f60 con 0x7f23280ce830 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232e400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23280b6de0 con 0x7f232815d5a0 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232e400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23280b54b0 con 0x7f23280b1e80 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.968+0000 7f232d000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 0x7f23280c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 0x7f23280c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58666/0 (socket says 172.21.15.164:58666) 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 -- 172.21.15.164:0/2257717131 learned_addr learned my addr 172.21.15.164:0/2257717131 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232da00640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232815d5a0 0x7f232815f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f2327e00640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23280ce830 0x7f232815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 -- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232815d5a0 msgr2=0x7f232815f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232815d5a0 0x7f232815f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 -- 172.21.15.164:0/2257717131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23280ce830 msgr2=0x7f232815f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23280ce830 0x7f232815f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 -- 172.21.15.164:0/2257717131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232815fda0 con 0x7f23280b1e80 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.969+0000 7f232d000640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 0x7f23280c8570 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f231c006010 tx=0x7f231c0039e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c03e030 con 0x7f23280b1e80 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f231c0047a0 con 0x7f23280b1e80 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2328160080 con 0x7f23280b1e80 2024-07-26T06:13:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c035040 con 0x7f23280b1e80 2024-07-26T06:13:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2328160640 con 0x7f23280b1e80 2024-07-26T06:13:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f231c041030 con 0x7f23280b1e80 2024-07-26T06:13:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.970+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f231c043d10 con 0x7f23280b1e80 2024-07-26T06:13:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.971+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23280ce2f0 con 0x7f23280b1e80 2024-07-26T06:13:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:10.973+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f231c043410 con 0x7f23280b1e80 2024-07-26T06:13:11.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.051+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "69942df7-d9b8-4051-b4ec-422cf7beee9b", "id": 1} v 0) -- 0x7f232815cc90 con 0x7f23280b1e80 2024-07-26T06:13:11.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.057+0000 7f2324c00640 1 -- 172.21.15.164:0/2257717131 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69942df7-d9b8-4051-b4ec-422cf7beee9b", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f231c005120 con 0x7f23280b1e80 2024-07-26T06:13:11.115 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.058+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 msgr2=0x7f23280c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.058+0000 7f232e400640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 0x7f23280c8570 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f231c006010 tx=0x7f231c0039e0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 shutdown_connections 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232815d5a0 0x7f232815f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23280ce830 0x7f232815f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 --2- 172.21.15.164:0/2257717131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23280b1e80 0x7f23280c8570 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 >> 172.21.15.164:0/2257717131 conn(0x7f23280cfda0 msgr2=0x7f232815aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 shutdown_connections 2024-07-26T06:13:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.059+0000 7f232e400640 1 -- 172.21.15.164:0/2257717131 wait complete. 2024-07-26T06:13:11.118 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new ed5f78dd-777d-4cb0-a6d7-5f6442deba3a 2 2024-07-26T06:13:11.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 -- 172.21.15.164:0/117775166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed8160e60 msgr2=0x7fbed8140ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 --2- 172.21.15.164:0/117775166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed8160e60 0x7fbed8140ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbec4005fd0 tx=0x7fbec402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 -- 172.21.15.164:0/117775166 shutdown_connections 2024-07-26T06:13:11.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 --2- 172.21.15.164:0/117775166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed8160e60 0x7fbed8140ac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 -- 172.21.15.164:0/117775166 >> 172.21.15.164:0/117775166 conn(0x7fbed80cfa50 msgr2=0x7fbed80cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 -- 172.21.15.164:0/117775166 shutdown_connections 2024-07-26T06:13:11.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.297+0000 7fbede600640 1 -- 172.21.15.164:0/117775166 wait complete. 2024-07-26T06:13:11.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.299+0000 7fbede600640 1 Processor -- start 2024-07-26T06:13:11.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.299+0000 7fbede600640 1 -- start start 2024-07-26T06:13:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.300+0000 7fbede600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbed80729f0 0x7fbed815c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbede600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 0x7fbed81f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbede600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed8160e60 0x7fbed80c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbede600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbed815e7d0 con 0x7fbed80729f0 2024-07-26T06:13:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbede600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbed815e650 con 0x7fbed80aee80 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbede600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbed8072e20 con 0x7fbed8160e60 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbeddc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed8160e60 0x7fbed80c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 0x7fbed81f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbeddc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed8160e60 0x7fbed80c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58672/0 (socket says 172.21.15.164:58672) 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 0x7fbed81f5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43708/0 (socket says 172.21.15.164:43708) 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbeddc00640 1 -- 172.21.15.164:0/347341435 learned_addr learned my addr 172.21.15.164:0/347341435 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbedd200640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbed80729f0 0x7fbed815c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 -- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed8160e60 msgr2=0x7fbed80c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed8160e60 0x7fbed80c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 -- 172.21.15.164:0/347341435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbed80729f0 msgr2=0x7fbed815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbed80729f0 0x7fbed815c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.301+0000 7fbed7e00640 1 -- 172.21.15.164:0/347341435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed81f4db0 con 0x7fbed80aee80 2024-07-26T06:13:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbed7e00640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 0x7fbed81f5010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbec4031000 tx=0x7fbec40038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec403e030 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbec4003ca0 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed80aa5c0 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4035040 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed80aab80 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.302+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbec4041030 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.303+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fbec4043cf0 con 0x7fbed80aee80 2024-07-26T06:13:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.303+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed80732e0 con 0x7fbed80aee80 2024-07-26T06:13:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.306+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fbed815d870 con 0x7fbed80aee80 2024-07-26T06:13:11.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.384+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ed5f78dd-777d-4cb0-a6d7-5f6442deba3a", "id": 2} v 0) -- 0x7fbed8073760 con 0x7fbed80aee80 2024-07-26T06:13:11.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.391+0000 7fbed4c00640 1 -- 172.21.15.164:0/347341435 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ed5f78dd-777d-4cb0-a6d7-5f6442deba3a", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fbec4043410 con 0x7fbed80aee80 2024-07-26T06:13:11.393 INFO:teuthology.orchestra.run.smithi164.stdout:2 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 msgr2=0x7fbed81f5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 0x7fbed81f5010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbec4031000 tx=0x7fbec40038a0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 shutdown_connections 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed8160e60 0x7fbed80c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed80aee80 0x7fbed81f5010 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 --2- 172.21.15.164:0/347341435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbed80729f0 0x7fbed815c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.392+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 >> 172.21.15.164:0/347341435 conn(0x7fbed80cfa50 msgr2=0x7fbed80b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.393+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 shutdown_connections 2024-07-26T06:13:11.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.393+0000 7fbede600640 1 -- 172.21.15.164:0/347341435 wait complete. 2024-07-26T06:13:11.411 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 1f2fe985-5805-4253-bd18-d1b35e1f5dcf 3 2024-07-26T06:13:11.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 -- 172.21.15.164:0/1837868837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf815d5a0 msgr2=0x7fbdf80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/1837868837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf815d5a0 0x7fbdf80ce2f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbde8005fd0 tx=0x7fbde802a410 comp rx=0 tx=0).stop 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 -- 172.21.15.164:0/1837868837 shutdown_connections 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/1837868837 >> v2:172.21.15.164:3300/0 conn(0x7fbdf80ce830 0x7fbdf80b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/1837868837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf815d5a0 0x7fbdf80ce2f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 -- 172.21.15.164:0/1837868837 >> 172.21.15.164:0/1837868837 conn(0x7fbdf80cfda0 msgr2=0x7fbdf80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 -- 172.21.15.164:0/1837868837 shutdown_connections 2024-07-26T06:13:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.596+0000 7fbdf7e00640 1 -- 172.21.15.164:0/1837868837 wait complete. 2024-07-26T06:13:11.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.598+0000 7fbdf7e00640 1 Processor -- start 2024-07-26T06:13:11.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.598+0000 7fbdf7e00640 1 -- start start 2024-07-26T06:13:11.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 0x7fbdf81608f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdf80ce830 0x7fbdf80c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdf815d5a0 0x7fbdf8152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf80b5130 con 0x7fbdf80ce830 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf80b4fb0 con 0x7fbdf815d5a0 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf80b52b0 con 0x7fbdf80b1e80 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 0x7fbdf81608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 0x7fbdf81608f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58692/0 (socket says 172.21.15.164:58692) 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 -- 172.21.15.164:0/2528753879 learned_addr learned my addr 172.21.15.164:0/2528753879 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf7400640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdf815d5a0 0x7fbdf8152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 -- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdf815d5a0 msgr2=0x7fbdf8152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdf815d5a0 0x7fbdf8152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 -- 172.21.15.164:0/2528753879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdf80ce830 msgr2=0x7fbdf80c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdf80ce830 0x7fbdf80c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.600+0000 7fbdf6a00640 1 -- 172.21.15.164:0/2528753879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf81f3f00 con 0x7fbdf80b1e80 2024-07-26T06:13:11.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdf6a00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 0x7fbdf81608f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbde8034040 tx=0x7fbde80039e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde803e030 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbde80047a0 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde8035040 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf80c83a0 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf80c8920 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.601+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbde8041030 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.602+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fbde8043410 con 0x7fbdf80b1e80 2024-07-26T06:13:11.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.602+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf80ce2f0 con 0x7fbdf80b1e80 2024-07-26T06:13:11.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.604+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fbde8004f40 con 0x7fbdf80b1e80 2024-07-26T06:13:11.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.683+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1f2fe985-5805-4253-bd18-d1b35e1f5dcf", "id": 3} v 0) -- 0x7fbdf80d26d0 con 0x7fbdf80b1e80 2024-07-26T06:13:11.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.689+0000 7fbdeea00640 1 -- 172.21.15.164:0/2528753879 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f2fe985-5805-4253-bd18-d1b35e1f5dcf", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fbde80051a0 con 0x7fbdf80b1e80 2024-07-26T06:13:11.690 INFO:teuthology.orchestra.run.smithi164.stdout:3 2024-07-26T06:13:11.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 msgr2=0x7fbdf81608f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 0x7fbdf81608f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbde8034040 tx=0x7fbde80039e0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 shutdown_connections 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdf815d5a0 0x7fbdf8152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdf80ce830 0x7fbdf80c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 --2- 172.21.15.164:0/2528753879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdf80b1e80 0x7fbdf81608f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 >> 172.21.15.164:0/2528753879 conn(0x7fbdf80cfda0 msgr2=0x7fbdf815bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 shutdown_connections 2024-07-26T06:13:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.690+0000 7fbdf7e00640 1 -- 172.21.15.164:0/2528753879 wait complete. 2024-07-26T06:13:11.708 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 17f99c8c-2d53-49ac-b648-1cb8f6f36466 4 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 -- 172.21.15.164:0/3705311061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 msgr2=0x7f912c0ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 --2- 172.21.15.164:0/3705311061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 0x7f912c0ce2f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9120009900 tx=0x7f912002a4a0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 -- 172.21.15.164:0/3705311061 shutdown_connections 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 --2- 172.21.15.164:0/3705311061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 0x7f912c0ce2f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 -- 172.21.15.164:0/3705311061 >> 172.21.15.164:0/3705311061 conn(0x7f912c0cfda0 msgr2=0x7f912c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 -- 172.21.15.164:0/3705311061 shutdown_connections 2024-07-26T06:13:11.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.885+0000 7f9132e00640 1 -- 172.21.15.164:0/3705311061 wait complete. 2024-07-26T06:13:11.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.888+0000 7f9132e00640 1 Processor -- start 2024-07-26T06:13:11.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.888+0000 7f9132e00640 1 -- start start 2024-07-26T06:13:11.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.889+0000 7f9132e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 0x7f912c1f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.889+0000 7f9132e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f912c0ce830 0x7f912c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.889+0000 7f9132e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 0x7f912c0c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:11.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9132e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f912c0b6f60 con 0x7f912c0ce830 2024-07-26T06:13:11.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9132e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f912c0b6de0 con 0x7f912c0b1e80 2024-07-26T06:13:11.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9132e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f912c0b54b0 con 0x7f912c15d5a0 2024-07-26T06:13:11.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 0x7f912c1f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 0x7f912c1f4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43724/0 (socket says 172.21.15.164:43724) 2024-07-26T06:13:11.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 -- 172.21.15.164:0/1601261702 learned_addr learned my addr 172.21.15.164:0/1601261702 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:11.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9132400640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 0x7f912c0c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131000640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f912c0ce830 0x7f912c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:11.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 -- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 msgr2=0x7f912c0c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 0x7f912c0c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 -- 172.21.15.164:0/1601261702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f912c0ce830 msgr2=0x7f912c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f912c0ce830 0x7f912c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.890+0000 7f9131a00640 1 -- 172.21.15.164:0/1601261702 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c0c6e40 con 0x7f912c0b1e80 2024-07-26T06:13:11.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9131a00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 0x7f912c1f4dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9120006010 tx=0x7f91200039e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:11.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912003e030 con 0x7f912c0b1e80 2024-07-26T06:13:11.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91200047a0 con 0x7f912c0b1e80 2024-07-26T06:13:11.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120035040 con 0x7f912c0b1e80 2024-07-26T06:13:11.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912c0c7120 con 0x7f912c0b1e80 2024-07-26T06:13:11.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912c0c76e0 con 0x7f912c0b1e80 2024-07-26T06:13:11.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9120041030 con 0x7f912c0b1e80 2024-07-26T06:13:11.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.891+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f9120043410 con 0x7f912c0b1e80 2024-07-26T06:13:11.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.892+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f912c0ce2f0 con 0x7f912c0b1e80 2024-07-26T06:13:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.894+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f9120004f40 con 0x7f912c0b1e80 2024-07-26T06:13:11.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.973+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "17f99c8c-2d53-49ac-b648-1cb8f6f36466", "id": 4} v 0) -- 0x7f912c0682e0 con 0x7f912c0b1e80 2024-07-26T06:13:11.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.980+0000 7f9129600640 1 -- 172.21.15.164:0/1601261702 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17f99c8c-2d53-49ac-b648-1cb8f6f36466", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f9120005120 con 0x7f912c0b1e80 2024-07-26T06:13:11.981 INFO:teuthology.orchestra.run.smithi164.stdout:4 2024-07-26T06:13:11.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 msgr2=0x7f912c1f4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:11.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 0x7f912c1f4dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9120006010 tx=0x7f91200039e0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 shutdown_connections 2024-07-26T06:13:11.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f912c15d5a0 0x7f912c0c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f912c0ce830 0x7f912c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 --2- 172.21.15.164:0/1601261702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f912c0b1e80 0x7f912c1f4dc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:11.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 >> 172.21.15.164:0/1601261702 conn(0x7f912c0cfda0 msgr2=0x7f912c15c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:11.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 shutdown_connections 2024-07-26T06:13:11.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:11.982+0000 7f9132e00640 1 -- 172.21.15.164:0/1601261702 wait complete. 2024-07-26T06:13:11.999 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new edacd1f8-feda-4c3e-b41b-7ca4420d418e 5 2024-07-26T06:13:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 -- 172.21.15.164:0/4147165901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc06fc70 msgr2=0x7f5ddc06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 --2- 172.21.15.164:0/4147165901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc06fc70 0x7f5ddc06d380 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4005fd0 tx=0x7f5dd402aa90 comp rx=0 tx=0).stop 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 -- 172.21.15.164:0/4147165901 shutdown_connections 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 --2- 172.21.15.164:0/4147165901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc06fc70 0x7f5ddc06d380 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 --2- 172.21.15.164:0/4147165901 >> v2:172.21.15.164:3300/0 conn(0x7f5ddc073900 0x7f5ddc15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 -- 172.21.15.164:0/4147165901 >> 172.21.15.164:0/4147165901 conn(0x7f5ddc06bf40 msgr2=0x7f5ddc06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 -- 172.21.15.164:0/4147165901 shutdown_connections 2024-07-26T06:13:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.178+0000 7f5de1200640 1 -- 172.21.15.164:0/4147165901 wait complete. 2024-07-26T06:13:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.181+0000 7f5de1200640 1 Processor -- start 2024-07-26T06:13:12.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.181+0000 7f5de1200640 1 -- start start 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.182+0000 7f5de1200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 0x7f5ddc0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.182+0000 7f5de1200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc073520 0x7f5ddc15e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.182+0000 7f5de1200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ddc073900 0x7f5ddc15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.182+0000 7f5de1200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc06dc50 con 0x7f5ddc073900 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.182+0000 7f5de1200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc06dad0 con 0x7f5ddc06fc70 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.182+0000 7f5de1200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5ddc06ddd0 con 0x7f5ddc073520 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 0x7f5ddc0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5dd3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc073520 0x7f5ddc15e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 0x7f5ddc0c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44390/0 (socket says 172.21.15.164:44390) 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5dd3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc073520 0x7f5ddc15e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38438/0 (socket says 172.21.15.164:38438) 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 -- 172.21.15.164:0/3242059786 learned_addr learned my addr 172.21.15.164:0/3242059786 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddbe00640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ddc073900 0x7f5ddc15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 -- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc073520 msgr2=0x7f5ddc15e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc073520 0x7f5ddc15e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 -- 172.21.15.164:0/3242059786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ddc073900 msgr2=0x7f5ddc15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ddc073900 0x7f5ddc15ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 -- 172.21.15.164:0/3242059786 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc148950 con 0x7f5ddc06fc70 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.183+0000 7f5ddb400640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 0x7f5ddc0c6890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00c450 tx=0x7f5dcc00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc0052d0 con 0x7f5ddc06fc70 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dcc005470 con 0x7f5ddc06fc70 2024-07-26T06:13:12.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc1f48a0 con 0x7f5ddc06fc70 2024-07-26T06:13:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc012870 con 0x7f5ddc06fc70 2024-07-26T06:13:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc148f00 con 0x7f5ddc06fc70 2024-07-26T06:13:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5dcc012a10 con 0x7f5ddc06fc70 2024-07-26T06:13:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.184+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f5dcc003b70 con 0x7f5ddc06fc70 2024-07-26T06:13:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.185+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ddc06d380 con 0x7f5ddc06fc70 2024-07-26T06:13:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.189+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f5dcc0035f0 con 0x7f5ddc06fc70 2024-07-26T06:13:12.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.268+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "edacd1f8-feda-4c3e-b41b-7ca4420d418e", "id": 5} v 0) -- 0x7f5ddc15d480 con 0x7f5ddc06fc70 2024-07-26T06:13:12.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.273+0000 7f5dd3400640 1 -- 172.21.15.164:0/3242059786 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "edacd1f8-feda-4c3e-b41b-7ca4420d418e", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f5dcc0038c0 con 0x7f5ddc06fc70 2024-07-26T06:13:12.275 INFO:teuthology.orchestra.run.smithi164.stdout:5 2024-07-26T06:13:12.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 msgr2=0x7f5ddc0c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 0x7f5ddc0c6890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00c450 tx=0x7f5dcc00c910 comp rx=0 tx=0).stop 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 shutdown_connections 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5ddc073900 0x7f5ddc15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ddc073520 0x7f5ddc15e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 --2- 172.21.15.164:0/3242059786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ddc06fc70 0x7f5ddc0c6890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 >> 172.21.15.164:0/3242059786 conn(0x7f5ddc06bf40 msgr2=0x7f5ddc1570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.275+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 shutdown_connections 2024-07-26T06:13:12.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.276+0000 7f5de1200640 1 -- 172.21.15.164:0/3242059786 wait complete. 2024-07-26T06:13:12.295 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 1ddea884-310a-4d70-8dbe-d2c9f30f2436 6 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 -- 172.21.15.164:0/3589293748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d015b0c0 msgr2=0x7f96d00b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 --2- 172.21.15.164:0/3589293748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d015b0c0 0x7f96d00b4f70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f96c0005fd0 tx=0x7f96c002a540 comp rx=0 tx=0).stop 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 -- 172.21.15.164:0/3589293748 shutdown_connections 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 --2- 172.21.15.164:0/3589293748 >> v2:172.21.15.163:3300/0 conn(0x7f96d0070a20 0x7f96d0070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 --2- 172.21.15.164:0/3589293748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d015b0c0 0x7f96d00b4f70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 -- 172.21.15.164:0/3589293748 >> 172.21.15.164:0/3589293748 conn(0x7f96d00b3060 msgr2=0x7f96d00b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.468+0000 7f96d5400640 1 -- 172.21.15.164:0/3589293748 shutdown_connections 2024-07-26T06:13:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.469+0000 7f96d5400640 1 -- 172.21.15.164:0/3589293748 wait complete. 2024-07-26T06:13:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.470+0000 7f96d5400640 1 Processor -- start 2024-07-26T06:13:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.470+0000 7f96d5400640 1 -- start start 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96d0070a20 0x7f96d0159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d5400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 0x7f96d0159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d5400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96d015b0c0 0x7f96d01499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d5400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f96d00b5d10 con 0x7f96d0070a20 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d5400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96d00b5b90 con 0x7f96d015b0c0 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d5400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96d00b5e90 con 0x7f96d00cff20 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96cf400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 0x7f96d0159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96d4a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96d015b0c0 0x7f96d01499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96cf400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 0x7f96d0159ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38462/0 (socket says 172.21.15.164:38462) 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.471+0000 7f96cf400640 1 -- 172.21.15.164:0/1121611547 learned_addr learned my addr 172.21.15.164:0/1121611547 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cfe00640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96d0070a20 0x7f96d0159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cf400640 1 -- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96d015b0c0 msgr2=0x7f96d01499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cf400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96d015b0c0 0x7f96d01499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cf400640 1 -- 172.21.15.164:0/1121611547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96d0070a20 msgr2=0x7f96d0159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cf400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96d0070a20 0x7f96d0159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cf400640 1 -- 172.21.15.164:0/1121611547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d0147800 con 0x7f96d00cff20 2024-07-26T06:13:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96cf400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 0x7f96d0159ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00c9a0 tx=0x7f96bc00ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc004220 con 0x7f96d00cff20 2024-07-26T06:13:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96bc0044b0 con 0x7f96d00cff20 2024-07-26T06:13:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc007c10 con 0x7f96d00cff20 2024-07-26T06:13:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.472+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d0147aa0 con 0x7f96d00cff20 2024-07-26T06:13:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.473+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d0148060 con 0x7f96d00cff20 2024-07-26T06:13:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.473+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f96bc0039a0 con 0x7f96d00cff20 2024-07-26T06:13:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.473+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f96bc007db0 con 0x7f96d00cff20 2024-07-26T06:13:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.474+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d00b63b0 con 0x7f96d00cff20 2024-07-26T06:13:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.477+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f96bc014030 con 0x7f96d00cff20 2024-07-26T06:13:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.556+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1ddea884-310a-4d70-8dbe-d2c9f30f2436", "id": 6} v 0) -- 0x7f96d015d620 con 0x7f96d00cff20 2024-07-26T06:13:12.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.563+0000 7f96c7e00640 1 -- 172.21.15.164:0/1121611547 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ddea884-310a-4d70-8dbe-d2c9f30f2436", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f96bc003dc0 con 0x7f96d00cff20 2024-07-26T06:13:12.564 INFO:teuthology.orchestra.run.smithi164.stdout:6 2024-07-26T06:13:12.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.564+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 msgr2=0x7f96d0159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.564+0000 7f96d5400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 0x7f96d0159ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00c9a0 tx=0x7f96bc00ce60 comp rx=0 tx=0).stop 2024-07-26T06:13:12.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 shutdown_connections 2024-07-26T06:13:12.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96d015b0c0 0x7f96d01499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96d00cff20 0x7f96d0159ad0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 --2- 172.21.15.164:0/1121611547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96d0070a20 0x7f96d0159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 >> 172.21.15.164:0/1121611547 conn(0x7f96d00b3060 msgr2=0x7f96d00b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:12.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 shutdown_connections 2024-07-26T06:13:12.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.565+0000 7f96d5400640 1 -- 172.21.15.164:0/1121611547 wait complete. 2024-07-26T06:13:12.584 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 0d239eae-64f6-47e6-a731-2e29aceb30ff 7 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.766+0000 7fe62e600640 1 -- 172.21.15.164:0/2135217194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 msgr2=0x7fe62815c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.766+0000 7fe62e600640 1 --2- 172.21.15.164:0/2135217194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe62815c6d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe620009830 tx=0x7fe62002a510 comp rx=0 tx=0).stop 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.767+0000 7fe62e600640 1 -- 172.21.15.164:0/2135217194 shutdown_connections 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.767+0000 7fe62e600640 1 --2- 172.21.15.164:0/2135217194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe62815c6d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.767+0000 7fe62e600640 1 --2- 172.21.15.164:0/2135217194 >> v2:172.21.15.163:3300/0 conn(0x7fe62815f690 0x7fe6280729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.767+0000 7fe62e600640 1 -- 172.21.15.164:0/2135217194 >> 172.21.15.164:0/2135217194 conn(0x7fe62806d050 msgr2=0x7fe62806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.767+0000 7fe62e600640 1 -- 172.21.15.164:0/2135217194 shutdown_connections 2024-07-26T06:13:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.767+0000 7fe62e600640 1 -- 172.21.15.164:0/2135217194 wait complete. 2024-07-26T06:13:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.768+0000 7fe62e600640 1 Processor -- start 2024-07-26T06:13:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.768+0000 7fe62e600640 1 -- start start 2024-07-26T06:13:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.769+0000 7fe62e600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe628156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.769+0000 7fe62e600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6280d0750 0x7fe6280acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.769+0000 7fe62e600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe62815f690 0x7fe6280ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.769+0000 7fe62e600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe62815cfa0 con 0x7fe6280d0750 2024-07-26T06:13:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.769+0000 7fe62e600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe62815ce20 con 0x7fe62815f690 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.769+0000 7fe62e600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe6280736b0 con 0x7fe628072ee0 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe628156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe628156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38482/0 (socket says 172.21.15.164:38482) 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62dc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe62815f690 0x7fe6280ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 -- 172.21.15.164:0/1206732915 learned_addr learned my addr 172.21.15.164:0/1206732915 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe61fe00640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6280d0750 0x7fe6280acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 -- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe62815f690 msgr2=0x7fe6280ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe62815f690 0x7fe6280ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 -- 172.21.15.164:0/1206732915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6280d0750 msgr2=0x7fe6280acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6280d0750 0x7fe6280acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 -- 172.21.15.164:0/1206732915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6280ad5c0 con 0x7fe628072ee0 2024-07-26T06:13:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.770+0000 7fe62d200640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe628156f70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe61800cc50 tx=0x7fe61800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618016020 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe618012420 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61800ce60 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6280a9670 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6281574b0 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe618019020 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.771+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe618012be0 con 0x7fe628072ee0 2024-07-26T06:13:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.772+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62815c6d0 con 0x7fe628072ee0 2024-07-26T06:13:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.776+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fe618020020 con 0x7fe628072ee0 2024-07-26T06:13:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.855+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0d239eae-64f6-47e6-a731-2e29aceb30ff", "id": 7} v 0) -- 0x7fe628068e90 con 0x7fe628072ee0 2024-07-26T06:13:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.862+0000 7fe625600640 1 -- 172.21.15.164:0/1206732915 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d239eae-64f6-47e6-a731-2e29aceb30ff", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe62815fb20 con 0x7fe628072ee0 2024-07-26T06:13:12.863 INFO:teuthology.orchestra.run.smithi164.stdout:7 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.863+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 msgr2=0x7fe628156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.863+0000 7fe62e600640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe628156f70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe61800cc50 tx=0x7fe61800a430 comp rx=0 tx=0).stop 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 shutdown_connections 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe62815f690 0x7fe6280ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6280d0750 0x7fe6280acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 --2- 172.21.15.164:0/1206732915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe628072ee0 0x7fe628156f70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 >> 172.21.15.164:0/1206732915 conn(0x7fe62806d050 msgr2=0x7fe62806c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 shutdown_connections 2024-07-26T06:13:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:12.864+0000 7fe62e600640 1 -- 172.21.15.164:0/1206732915 wait complete. 2024-07-26T06:13:12.883 INFO:tasks.ceph.osd.0:Restarting daemon 2024-07-26T06:13:12.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-07-26T06:13:12.892 INFO:tasks.ceph.osd.0:Started 2024-07-26T06:13:12.893 INFO:tasks.ceph.osd.1:Restarting daemon 2024-07-26T06:13:12.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-07-26T06:13:12.900 INFO:tasks.ceph.osd.1:Started 2024-07-26T06:13:12.901 INFO:tasks.ceph.osd.2:Restarting daemon 2024-07-26T06:13:12.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-07-26T06:13:12.908 INFO:tasks.ceph.osd.2:Started 2024-07-26T06:13:12.909 INFO:tasks.ceph.osd.3:Restarting daemon 2024-07-26T06:13:12.909 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-07-26T06:13:12.914 INFO:tasks.ceph.osd.3:Started 2024-07-26T06:13:12.914 INFO:tasks.ceph.osd.4:Restarting daemon 2024-07-26T06:13:12.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-07-26T06:13:12.922 INFO:tasks.ceph.osd.4:Started 2024-07-26T06:13:12.922 INFO:tasks.ceph.osd.5:Restarting daemon 2024-07-26T06:13:12.922 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-07-26T06:13:12.929 INFO:tasks.ceph.osd.5:Started 2024-07-26T06:13:12.929 INFO:tasks.ceph.osd.6:Restarting daemon 2024-07-26T06:13:12.929 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-07-26T06:13:12.934 INFO:tasks.ceph.osd.6:Started 2024-07-26T06:13:12.934 INFO:tasks.ceph.osd.7:Restarting daemon 2024-07-26T06:13:12.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-07-26T06:13:12.940 INFO:tasks.ceph.osd.7:Started 2024-07-26T06:13:12.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:13.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 -- 172.21.15.163:0/241320115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c0b53b0 msgr2=0x7fec7c0b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:13.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 --2- 172.21.15.163:0/241320115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c0b53b0 0x7fec7c0b5800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec7c075ba0 tx=0x7fec6802a5f0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 -- 172.21.15.163:0/241320115 shutdown_connections 2024-07-26T06:13:13.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 --2- 172.21.15.163:0/241320115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c0b53b0 0x7fec7c0b5800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 --2- 172.21.15.163:0/241320115 >> v2:172.21.15.164:3300/0 conn(0x7fec7c15d650 0x7fec7c0d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 -- 172.21.15.163:0/241320115 >> 172.21.15.163:0/241320115 conn(0x7fec7c0d36b0 msgr2=0x7fec7c0d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:13.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 -- 172.21.15.163:0/241320115 shutdown_connections 2024-07-26T06:13:13.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.160+0000 7fec81e00640 1 -- 172.21.15.163:0/241320115 wait complete. 2024-07-26T06:13:13.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.162+0000 7fec81e00640 1 Processor -- start 2024-07-26T06:13:13.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.163+0000 7fec81e00640 1 -- start start 2024-07-26T06:13:13.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec81e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 0x7fec7c15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:13.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec81e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec7c0b53b0 0x7fec7c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:13.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec81e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec7c15d650 0x7fec7c15ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec81e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c0d2050 con 0x7fec7c06c270 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec81e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c0d1ed0 con 0x7fec7c15d650 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec81e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec7c0b7730 con 0x7fec7c0b53b0 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec7aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 0x7fec7c15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec7aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 0x7fec7c15f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45492/0 (socket says 172.21.15.163:45492) 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec7aa00640 1 -- 172.21.15.163:0/3057690913 learned_addr learned my addr 172.21.15.163:0/3057690913 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.164+0000 7fec7a000640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec7c0b53b0 0x7fec7c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7b400640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec7c15d650 0x7fec7c15ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7aa00640 1 -- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec7c0b53b0 msgr2=0x7fec7c15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7aa00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec7c0b53b0 0x7fec7c15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7aa00640 1 -- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec7c15d650 msgr2=0x7fec7c15ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7aa00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec7c15d650 0x7fec7c15ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7aa00640 1 -- 172.21.15.163:0/3057690913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7c149660 con 0x7fec7c06c270 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec7aa00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 0x7fec7c15f9f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00d740 tx=0x7fec6c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c017070 con 0x7fec7c06c270 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec6c009d20 con 0x7fec7c06c270 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0106c0 con 0x7fec7c06c270 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec7c158710 con 0x7fec7c06c270 2024-07-26T06:13:13.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.165+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec7c149bb0 con 0x7fec7c06c270 2024-07-26T06:13:13.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.167+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fec6c01a070 con 0x7fec7c06c270 2024-07-26T06:13:13.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.167+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fec6c01cda0 con 0x7fec7c06c270 2024-07-26T06:13:13.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.167+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec7c0b5800 con 0x7fec7c06c270 2024-07-26T06:13:13.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.170+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fec6c01c6d0 con 0x7fec7c06c270 2024-07-26T06:13:13.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.236+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec7c15ce90 con 0x7fec7c06c270 2024-07-26T06:13:13.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.237+0000 7fec73400640 1 -- 172.21.15.163:0/3057690913 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fec6c01c8b0 con 0x7fec7c06c270 2024-07-26T06:13:13.237 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:13.237 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":10,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:12.857980+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:13.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 msgr2=0x7fec7c15f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:13.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 0x7fec7c15f9f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00d740 tx=0x7fec6c00dc00 comp rx=0 tx=0).stop 2024-07-26T06:13:13.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 shutdown_connections 2024-07-26T06:13:13.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec7c15d650 0x7fec7c15ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec7c0b53b0 0x7fec7c15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 --2- 172.21.15.163:0/3057690913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fec7c06c270 0x7fec7c15f9f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:13.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 >> 172.21.15.163:0/3057690913 conn(0x7fec7c0d36b0 msgr2=0x7fec7c0ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:13.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 shutdown_connections 2024-07-26T06:13:13.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:13.238+0000 7fec81e00640 1 -- 172.21.15.163:0/3057690913 wait complete. 2024-07-26T06:13:13.253 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-07-26T06:13:13.253 INFO:tasks.ceph:Waiting for OSDs to come up 2024-07-26T06:13:13.587 INFO:tasks.ceph.osd.0.smithi163.stderr:2024-07-26T06:13:13.587+0000 7f220fcf0640 -1 Falling back to public interface 2024-07-26T06:13:13.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-07-26T06:13:13.605+0000 7ff4834fd640 -1 Falling back to public interface 2024-07-26T06:13:13.612 INFO:tasks.ceph.osd.2.smithi163.stderr:2024-07-26T06:13:13.612+0000 7fed66263640 -1 Falling back to public interface 2024-07-26T06:13:13.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-07-26T06:13:13.612+0000 7fce8b142640 -1 Falling back to public interface 2024-07-26T06:13:13.617 INFO:tasks.ceph.osd.1.smithi163.stderr:2024-07-26T06:13:13.618+0000 7ff253427640 -1 Falling back to public interface 2024-07-26T06:13:13.630 INFO:tasks.ceph.osd.3.smithi163.stderr:2024-07-26T06:13:13.630+0000 7f7c14213640 -1 Falling back to public interface 2024-07-26T06:13:13.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-07-26T06:13:13.639+0000 7f8a39229640 -1 Falling back to public interface 2024-07-26T06:13:13.652 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-07-26T06:13:13.651+0000 7f2672e43640 -1 Falling back to public interface 2024-07-26T06:13:14.061 DEBUG:teuthology.orchestra.run.smithi163:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:14.225 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 -- 172.21.15.163:0/3439415994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 msgr2=0x7f6c1406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:14.226 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 --2- 172.21.15.163:0/3439415994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c1406ccc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009830 tx=0x7f6c0002a530 comp rx=0 tx=0).stop 2024-07-26T06:13:14.226 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 -- 172.21.15.163:0/3439415994 shutdown_connections 2024-07-26T06:13:14.226 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 --2- 172.21.15.163:0/3439415994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c1406ccc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.226 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 -- 172.21.15.163:0/3439415994 >> 172.21.15.163:0/3439415994 conn(0x7f6c140d0060 msgr2=0x7f6c140d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:14.226 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 -- 172.21.15.163:0/3439415994 shutdown_connections 2024-07-26T06:13:14.227 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.224+0000 7f6c1bc00640 1 -- 172.21.15.163:0/3439415994 wait complete. 2024-07-26T06:13:14.232 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.226+0000 7f6c1bc00640 1 Processor -- start 2024-07-26T06:13:14.232 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.226+0000 7f6c1bc00640 1 -- start start 2024-07-26T06:13:14.232 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.227+0000 7f6c1bc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c14154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:14.232 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.227+0000 7f6c1bc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c1415d6b0 0x7f6c14155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:14.232 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.227+0000 7f6c1bc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c141618c0 0x7f6c1415dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:14.233 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.227+0000 7f6c1bc00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6c140d27b0 con 0x7f6c140b6940 2024-07-26T06:13:14.233 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.227+0000 7f6c1bc00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14161f60 con 0x7f6c141618c0 2024-07-26T06:13:14.233 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.227+0000 7f6c1bc00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6c140d2930 con 0x7f6c1415d6b0 2024-07-26T06:13:14.233 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c14154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:14.233 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c14154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45498/0 (socket says 172.21.15.163:45498) 2024-07-26T06:13:14.234 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 -- 172.21.15.163:0/1514489928 learned_addr learned my addr 172.21.15.163:0/1514489928 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:14.234 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c13e00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c1415d6b0 0x7f6c14155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:14.234 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19c00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c141618c0 0x7f6c1415dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:14.234 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 -- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c1415d6b0 msgr2=0x7f6c14155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:14.234 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c1415d6b0 0x7f6c14155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.235 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 -- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c141618c0 msgr2=0x7f6c1415dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:14.235 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c141618c0 0x7f6c1415dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.235 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 -- 172.21.15.163:0/1514489928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c141535c0 con 0x7f6c140b6940 2024-07-26T06:13:14.235 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c13e00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c1415d6b0 0x7f6c14155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:14.235 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.228+0000 7f6c19200640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c14154640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c410 tx=0x7f6c0800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:14.236 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.229+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08017070 con 0x7f6c140b6940 2024-07-26T06:13:14.236 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.229+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c080035f0 con 0x7f6c140b6940 2024-07-26T06:13:14.236 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.229+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08003a80 con 0x7f6c140b6940 2024-07-26T06:13:14.236 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.229+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c141538a0 con 0x7f6c140b6940 2024-07-26T06:13:14.236 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.229+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c14153e60 con 0x7f6c140b6940 2024-07-26T06:13:14.236 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.230+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c140d2e60 con 0x7f6c140b6940 2024-07-26T06:13:14.237 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.230+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 2) ==== 49122+0+0 (secure 0 0 0) 0x7f6c0801a070 con 0x7f6c140b6940 2024-07-26T06:13:14.237 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.231+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6c0804fd70 con 0x7f6c140b6940 2024-07-26T06:13:14.237 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.233+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f6c08033030 con 0x7f6c140b6940 2024-07-26T06:13:14.298 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.298+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c140742f0 con 0x7f6c140b6940 2024-07-26T06:13:14.298 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.299+0000 7f6c10c00640 1 -- 172.21.15.163:0/1514489928 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f6c0801ada0 con 0x7f6c140b6940 2024-07-26T06:13:14.299 INFO:teuthology.misc.health.smithi163.stdout: 2024-07-26T06:13:14.299 INFO:teuthology.misc.health.smithi163.stdout:{"epoch":10,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:12.857980+0000","last_up_change":"0.000000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 msgr2=0x7f6c14154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c14154640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c410 tx=0x7f6c0800c8d0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 shutdown_connections 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c141618c0 0x7f6c1415dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c1415d6b0 0x7f6c14155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 --2- 172.21.15.163:0/1514489928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c140b6940 0x7f6c14154640 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:14.300 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 >> 172.21.15.163:0/1514489928 conn(0x7f6c140d0060 msgr2=0x7f6c140d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:14.301 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 shutdown_connections 2024-07-26T06:13:14.301 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:14.300+0000 7f6c1bc00640 1 -- 172.21.15.163:0/1514489928 wait complete. 2024-07-26T06:13:14.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-07-26T06:13:14.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:13:14.666+0000 7f6d1d400640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-07-26T06:13:14.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:13:14.667+0000 7f6d1d400640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-07-26T06:13:17.262 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-07-26T06:13:17.261+0000 7f2672e43640 -1 osd.4 0 log_to_monitors true 2024-07-26T06:13:17.268 INFO:tasks.ceph.osd.1.smithi163.stderr:2024-07-26T06:13:17.268+0000 7ff253427640 -1 osd.1 0 log_to_monitors true 2024-07-26T06:13:17.269 INFO:tasks.ceph.osd.3.smithi163.stderr:2024-07-26T06:13:17.269+0000 7f7c14213640 -1 osd.3 0 log_to_monitors true 2024-07-26T06:13:17.426 INFO:tasks.ceph.osd.0.smithi163.stderr:2024-07-26T06:13:17.423+0000 7f220fcf0640 -1 osd.0 0 log_to_monitors true 2024-07-26T06:13:17.443 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-07-26T06:13:17.442+0000 7ff4834fd640 -1 osd.5 0 log_to_monitors true 2024-07-26T06:13:17.508 INFO:tasks.ceph.osd.2.smithi163.stderr:2024-07-26T06:13:17.508+0000 7fed66263640 -1 osd.2 0 log_to_monitors true 2024-07-26T06:13:17.527 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-07-26T06:13:17.526+0000 7fce8b142640 -1 osd.6 0 log_to_monitors true 2024-07-26T06:13:17.539 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-07-26T06:13:17.538+0000 7f8a39229640 -1 osd.7 0 log_to_monitors true 2024-07-26T06:13:19.689 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-07-26T06:13:19.688+0000 7f266dc00640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T06:13:19.762 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-07-26T06:13:19.761+0000 7f2663c00640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:19.963 INFO:tasks.ceph.osd.3.smithi163.stderr:2024-07-26T06:13:19.962+0000 7f7c0ea00640 -1 osd.3 0 waiting for initial osdmap 2024-07-26T06:13:19.982 INFO:tasks.ceph.osd.3.smithi163.stderr:2024-07-26T06:13:19.981+0000 7f7c04a00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:19.993 INFO:tasks.ceph.osd.1.smithi163.stderr:2024-07-26T06:13:19.992+0000 7ff24e200640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T06:13:20.008 INFO:tasks.ceph.osd.1.smithi163.stderr:2024-07-26T06:13:20.007+0000 7ff244200640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:20.814 INFO:tasks.ceph.osd.2.smithi163.stderr:2024-07-26T06:13:20.814+0000 7fed61200640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T06:13:20.818 INFO:tasks.ceph.osd.0.smithi163.stderr:2024-07-26T06:13:20.818+0000 7f220b600640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T06:13:20.820 INFO:tasks.ceph.osd.2.smithi163.stderr:2024-07-26T06:13:20.819+0000 7fed57200640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:20.824 INFO:tasks.ceph.osd.0.smithi163.stderr:2024-07-26T06:13:20.823+0000 7f2200c00640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:21.120 DEBUG:teuthology.orchestra.run.smithi163:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:21.175 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-07-26T06:13:21.174+0000 7f8a34a00640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T06:13:21.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-07-26T06:13:21.185+0000 7f8a2a000640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:21.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-07-26T06:13:21.253+0000 7ff47e400640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T06:13:21.260 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-07-26T06:13:21.259+0000 7ff474400640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:21.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-07-26T06:13:21.299+0000 7fce86000640 -1 osd.6 0 waiting for initial osdmap 2024-07-26T06:13:21.309 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-07-26T06:13:21.308+0000 7fce7c000640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:13:21.313 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 -- 172.21.15.163:0/2167879454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640d1c40 msgr2=0x7f226415f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:21.313 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 --2- 172.21.15.163:0/2167879454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640d1c40 0x7f226415f520 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f224c005fd0 tx=0x7f224c02a540 comp rx=0 tx=0).stop 2024-07-26T06:13:21.313 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 -- 172.21.15.163:0/2167879454 shutdown_connections 2024-07-26T06:13:21.313 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 --2- 172.21.15.163:0/2167879454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22640708c0 0x7f2264070cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.314 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 --2- 172.21.15.163:0/2167879454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640d1c40 0x7f226415f520 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.314 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 -- 172.21.15.163:0/2167879454 >> 172.21.15.163:0/2167879454 conn(0x7f22640d1120 msgr2=0x7f22640d1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:21.314 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 -- 172.21.15.163:0/2167879454 shutdown_connections 2024-07-26T06:13:21.314 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.312+0000 7f2268a00640 1 -- 172.21.15.163:0/2167879454 wait complete. 2024-07-26T06:13:21.315 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.314+0000 7f2268a00640 1 Processor -- start 2024-07-26T06:13:21.315 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.314+0000 7f2268a00640 1 -- start start 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.315+0000 7f2268a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 0x7f226406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.315+0000 7f2268a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22640b5450 0x7f22640ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.315+0000 7f2268a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22640d1c40 0x7f22640aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.315+0000 7f2268a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2264073360 con 0x7f22640708c0 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.315+0000 7f2268a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f22640731e0 con 0x7f22640b5450 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.315+0000 7f2268a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2264071210 con 0x7f22640d1c40 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 0x7f226406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2262a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22640b5450 0x7f22640ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 0x7f226406e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:57716/0 (socket says 172.21.15.163:57716) 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 -- 172.21.15.163:0/2260971187 learned_addr learned my addr 172.21.15.163:0/2260971187 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:21.318 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263e00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22640d1c40 0x7f22640aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 -- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22640d1c40 msgr2=0x7f22640aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22640d1c40 0x7f22640aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 -- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22640b5450 msgr2=0x7f22640ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22640b5450 0x7f22640ae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 -- 172.21.15.163:0/2260971187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22640af4d0 con 0x7f22640708c0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.316+0000 7f2263400640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 0x7f226406e1b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f225400c450 tx=0x7f225400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.317+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22540052d0 con 0x7f22640708c0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.317+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2254005470 con 0x7f22640708c0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.317+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254012870 con 0x7f22640708c0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.317+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22641527a0 con 0x7f22640708c0 2024-07-26T06:13:21.319 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.317+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2264152d60 con 0x7f22640708c0 2024-07-26T06:13:21.320 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.318+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22640716b0 con 0x7f22640708c0 2024-07-26T06:13:21.320 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.320+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f22540035f0 con 0x7f22640708c0 2024-07-26T06:13:21.321 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.321+0000 7f225b400640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2240076240 0x7f2240078700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:21.321 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.321+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3071+0+0 (secure 0 0 0) 0x7f2254096690 con 0x7f22640708c0 2024-07-26T06:13:21.322 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.321+0000 7f2262a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2240076240 0x7f2240078700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:21.322 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.322+0000 7f2262a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2240076240 0x7f2240078700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f224c033000 tx=0x7f224c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:21.326 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.326+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22540604c0 con 0x7f22640708c0 2024-07-26T06:13:21.537 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.536+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22640719f0 con 0x7f22640708c0 2024-07-26T06:13:21.539 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.538+0000 7f225b400640 1 -- 172.21.15.163:0/2260971187 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+8527 (secure 0 0 0) 0x7f22540647c0 con 0x7f22640708c0 2024-07-26T06:13:21.539 INFO:teuthology.misc.health.smithi163.stdout: 2024-07-26T06:13:21.539 INFO:teuthology.misc.health.smithi163.stdout:{"epoch":14,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:20.734427+0000","last_up_change":"2024-07-26T06:13:20.734427+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:21.541 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.540+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2240076240 msgr2=0x7f2240078700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:21.541 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.540+0000 7f2268a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2240076240 0x7f2240078700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f224c033000 tx=0x7f224c004730 comp rx=0 tx=0).stop 2024-07-26T06:13:21.541 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 msgr2=0x7f226406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:21.542 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 0x7f226406e1b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f225400c450 tx=0x7f225400c910 comp rx=0 tx=0).stop 2024-07-26T06:13:21.542 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 shutdown_connections 2024-07-26T06:13:21.542 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2240076240 0x7f2240078700 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.542 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22640d1c40 0x7f22640aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.542 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22640b5450 0x7f22640ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.542 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 --2- 172.21.15.163:0/2260971187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22640708c0 0x7f226406e1b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:21.543 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 >> 172.21.15.163:0/2260971187 conn(0x7f22640d1120 msgr2=0x7f22640b1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:21.543 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 shutdown_connections 2024-07-26T06:13:21.543 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:21.541+0000 7f2268a00640 1 -- 172.21.15.163:0/2260971187 wait complete. 2024-07-26T06:13:21.556 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-07-26T06:13:25.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:13:25.617 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:13:28.363 DEBUG:teuthology.orchestra.run.smithi163:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:28.555 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 -- 172.21.15.163:0/320962066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d240ce5b0 msgr2=0x7f0d240d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:28.555 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 --2- 172.21.15.163:0/320962066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d240ce5b0 0x7f0d240d3330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d240cd9a0 tx=0x7f0d0c02a560 comp rx=0 tx=0).stop 2024-07-26T06:13:28.555 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 -- 172.21.15.163:0/320962066 shutdown_connections 2024-07-26T06:13:28.555 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 --2- 172.21.15.163:0/320962066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d240d3870 0x7f0d240b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.555 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 --2- 172.21.15.163:0/320962066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d240ce5b0 0x7f0d240d3330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.555 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 --2- 172.21.15.163:0/320962066 >> v2:172.21.15.164:3300/0 conn(0x7f0d24160910 0x7f0d240d2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.556 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 -- 172.21.15.163:0/320962066 >> 172.21.15.163:0/320962066 conn(0x7f0d2406c1d0 msgr2=0x7f0d2406c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:28.556 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 -- 172.21.15.163:0/320962066 shutdown_connections 2024-07-26T06:13:28.556 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.554+0000 7f0d2a400640 1 -- 172.21.15.163:0/320962066 wait complete. 2024-07-26T06:13:28.556 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.555+0000 7f0d2a400640 1 Processor -- start 2024-07-26T06:13:28.556 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.556+0000 7f0d2a400640 1 -- start start 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.556+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d240ce5b0 0x7f0d240aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d240d3870 0x7f0d241f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d2a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 0x7f0d24154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d2a400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0d240b54e0 con 0x7f0d24160910 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d2a400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0d240b5360 con 0x7f0d240d3870 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d2a400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0d240b5660 con 0x7f0d240ce5b0 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 0x7f0d24154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 0x7f0d24154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49288/0 (socket says 172.21.15.163:49288) 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 -- 172.21.15.163:0/43634511 learned_addr learned my addr 172.21.15.163:0/43634511 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29000640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d240ce5b0 0x7f0d240aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d1be00640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d240d3870 0x7f0d241f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:28.559 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 -- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d240ce5b0 msgr2=0x7f0d240aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d240ce5b0 0x7f0d240aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 -- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d240d3870 msgr2=0x7f0d241f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d240d3870 0x7f0d241f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d29a00640 1 -- 172.21.15.163:0/43634511 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d241f4cd0 con 0x7f0d24160910 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.557+0000 7f0d1be00640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d240d3870 0x7f0d241f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.558+0000 7f0d29a00640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 0x7f0d24154720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00ca40 tx=0x7f0d1c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.558+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c00adc0 con 0x7f0d24160910 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.558+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d1c004c50 con 0x7f0d24160910 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.558+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c005280 con 0x7f0d24160910 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.558+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d240a9ed0 con 0x7f0d24160910 2024-07-26T06:13:28.560 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.558+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d240aa490 con 0x7f0d24160910 2024-07-26T06:13:28.561 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.559+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d240b5bf0 con 0x7f0d24160910 2024-07-26T06:13:28.561 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.560+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0d1c005420 con 0x7f0d24160910 2024-07-26T06:13:28.562 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.561+0000 7f0d21600640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0d000762d0 0x7f0d00078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:28.562 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.561+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0d1c097b90 con 0x7f0d24160910 2024-07-26T06:13:28.562 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.562+0000 7f0d29000640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0d000762d0 0x7f0d00078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:28.563 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.563+0000 7f0d29000640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0d000762d0 0x7f0d00078790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d140058c0 tx=0x7f0d1400c440 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:28.567 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.566+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d1c014030 con 0x7f0d24160910 2024-07-26T06:13:28.777 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.776+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d240d10c0 con 0x7f0d24160910 2024-07-26T06:13:28.779 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.778+0000 7f0d21600640 1 -- 172.21.15.163:0/43634511 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12908 (secure 0 0 0) 0x7f0d1c0617a0 con 0x7f0d24160910 2024-07-26T06:13:28.779 INFO:teuthology.misc.health.smithi163.stdout: 2024-07-26T06:13:28.779 INFO:teuthology.misc.health.smithi163.stdout:{"epoch":17,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:24.365364+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.781+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0d000762d0 msgr2=0x7f0d00078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.781+0000 7f0d2a400640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0d000762d0 0x7f0d00078790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d140058c0 tx=0x7f0d1400c440 comp rx=0 tx=0).stop 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.781+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 msgr2=0x7f0d24154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.781+0000 7f0d2a400640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 0x7f0d24154720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00ca40 tx=0x7f0d1c00cf00 comp rx=0 tx=0).stop 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 shutdown_connections 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0d000762d0 0x7f0d00078790 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0d24160910 0x7f0d24154720 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d240d3870 0x7f0d241f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 --2- 172.21.15.163:0/43634511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d240ce5b0 0x7f0d240aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 >> 172.21.15.163:0/43634511 conn(0x7f0d2406c1d0 msgr2=0x7f0d240b8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 shutdown_connections 2024-07-26T06:13:28.783 INFO:teuthology.misc.health.smithi163.stderr:2024-07-26T06:13:28.782+0000 7f0d2a400640 1 -- 172.21.15.163:0/43634511 wait complete. 2024-07-26T06:13:28.798 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-07-26T06:13:28.798 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-07-26T06:13:28.799 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:13:28.799 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:13:28.803 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:13:28.803 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:13:28.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:13:28.808 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:13:28.808 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:13:28.808 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:13:28.816 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:13:28.816 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:13:28.816 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:13:28.823 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:13:28.890 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-07-26T06:13:28.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:13:28.948 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:13:28.954 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:13:28.975 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:13:28.975 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:13:29.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.074+0000 7f00d4200640 1 -- 172.21.15.163:0/511880817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00b4003a50 msgr2=0x7f00b4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:29.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.074+0000 7f00d4200640 1 --2- 172.21.15.163:0/511880817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00b4003a50 0x7f00b4003e70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f00c000baf0 tx=0x7f00b4005830 comp rx=0 tx=0).stop 2024-07-26T06:13:29.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.074+0000 7f00d2200640 1 -- 172.21.15.163:0/511880817 reap_dead start 2024-07-26T06:13:29.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.075+0000 7f00d4200640 1 -- 172.21.15.163:0/511880817 shutdown_connections 2024-07-26T06:13:29.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.075+0000 7f00d4200640 1 -- 172.21.15.163:0/511880817 >> 172.21.15.163:0/511880817 conn(0x7f00cc06d420 msgr2=0x7f00cc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:29.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.075+0000 7f00d4200640 1 -- 172.21.15.163:0/511880817 shutdown_connections 2024-07-26T06:13:29.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.075+0000 7f00d4200640 1 -- 172.21.15.163:0/511880817 wait complete. 2024-07-26T06:13:29.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.076+0000 7f00d4200640 1 Processor -- start 2024-07-26T06:13:29.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.076+0000 7f00d4200640 1 -- start start 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.077+0000 7f00d4200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b4003a50 0x7f00cc14aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.077+0000 7f00d4200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 0x7f00cc0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.077+0000 7f00d4200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00cc0ade20 0x7f00cc0cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.077+0000 7f00d4200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc15eea0 con 0x7f00cc14cd60 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.077+0000 7f00d4200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc15ed20 con 0x7f00b4003a50 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.077+0000 7f00d4200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f00cc15f020 con 0x7f00cc0ade20 2024-07-26T06:13:29.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 0x7f00cc0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 0x7f00cc0ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49338/0 (socket says 172.21.15.163:49338) 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 -- 172.21.15.163:0/4030215280 learned_addr learned my addr 172.21.15.163:0/4030215280 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 -- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00cc0ade20 msgr2=0x7f00cc0cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d1800640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b4003a50 0x7f00cc14aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00cc0ade20 0x7f00cc0cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 -- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b4003a50 msgr2=0x7f00cc14aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b4003a50 0x7f00cc14aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 -- 172.21.15.163:0/4030215280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc14b0e0 con 0x7f00cc14cd60 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00d0e00640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 0x7f00cc0ad8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f00b800e630 tx=0x7f00b800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.078+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8015070 con 0x7f00cc14cd60 2024-07-26T06:13:29.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.079+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00b8004310 con 0x7f00cc14cd60 2024-07-26T06:13:29.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.079+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc0cca50 con 0x7f00cc14cd60 2024-07-26T06:13:29.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.079+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b800c9a0 con 0x7f00cc14cd60 2024-07-26T06:13:29.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.079+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc14c560 con 0x7f00cc14cd60 2024-07-26T06:13:29.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.080+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00cc0ae240 con 0x7f00cc14cd60 2024-07-26T06:13:29.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.081+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00b800cb40 con 0x7f00cc14cd60 2024-07-26T06:13:29.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.082+0000 7f00c9600640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00ac076300 0x7f00ac0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:29.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.082+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f00b8097820 con 0x7f00cc14cd60 2024-07-26T06:13:29.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.083+0000 7f00d1800640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00ac076300 0x7f00ac0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:29.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.084+0000 7f00d1800640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00ac076300 0x7f00ac0787c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f00c0036000 tx=0x7f00c00046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:29.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.087+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f00cc14b0e0 con 0x7f00cc14cd60 2024-07-26T06:13:29.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.329+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00cc15d210 con 0x7f00cc14cd60 2024-07-26T06:13:29.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.330+0000 7f00c9600640 1 -- 172.21.15.163:0/4030215280 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1437 (secure 0 0 0) 0x7f00b80614c0 con 0x7f00cc14cd60 2024-07-26T06:13:29.331 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:29.331 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":2,"btime":"2024-07-26T06:13:28:957673+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-26T06:13:29.331 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 2 2024-07-26T06:13:29.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00ac076300 msgr2=0x7f00ac0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:29.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00ac076300 0x7f00ac0787c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f00c0036000 tx=0x7f00c00046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 msgr2=0x7f00cc0ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:29.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 0x7f00cc0ad8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f00b800e630 tx=0x7f00b800eaf0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 shutdown_connections 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00ac076300 0x7f00ac0787c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00cc0ade20 0x7f00cc0cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00cc14cd60 0x7f00cc0ad8e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 --2- 172.21.15.163:0/4030215280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b4003a50 0x7f00cc14aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 >> 172.21.15.163:0/4030215280 conn(0x7f00cc06d420 msgr2=0x7f00cc0afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 shutdown_connections 2024-07-26T06:13:29.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:29.332+0000 7f00d4200640 1 -- 172.21.15.163:0/4030215280 wait complete. 2024-07-26T06:13:31.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:13:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 -- 172.21.15.163:0/3753992056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd27015d5a0 msgr2=0x7fd2700ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 --2- 172.21.15.163:0/3753992056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd27015d5a0 0x7fd2700ce2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd260005fd0 tx=0x7fd26002a4a0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 -- 172.21.15.163:0/3753992056 shutdown_connections 2024-07-26T06:13:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 --2- 172.21.15.163:0/3753992056 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2700b1e80 0x7fd2700b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 --2- 172.21.15.163:0/3753992056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd27015d5a0 0x7fd2700ce2f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 -- 172.21.15.163:0/3753992056 >> 172.21.15.163:0/3753992056 conn(0x7fd2700cfda0 msgr2=0x7fd2700d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 -- 172.21.15.163:0/3753992056 shutdown_connections 2024-07-26T06:13:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.567+0000 7fd26fe00640 1 -- 172.21.15.163:0/3753992056 wait complete. 2024-07-26T06:13:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.569+0000 7fd26fe00640 1 Processor -- start 2024-07-26T06:13:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.569+0000 7fd26fe00640 1 -- start start 2024-07-26T06:13:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 0x7fd27015fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2700ce830 0x7fd2700ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd27015d5a0 0x7fd2701f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd2700b5130 con 0x7fd2700b1e80 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd2700b4fb0 con 0x7fd2700ce830 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd2700b52b0 con 0x7fd27015d5a0 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 0x7fd27015fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.570+0000 7fd26ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 0x7fd27015fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49348/0 (socket says 172.21.15.163:49348) 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 -- 172.21.15.163:0/3178843967 learned_addr learned my addr 172.21.15.163:0/3178843967 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 -- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd27015d5a0 msgr2=0x7fd2701f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd27015d5a0 0x7fd2701f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 -- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2700ce830 msgr2=0x7fd2700ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2700ce830 0x7fd2700ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 -- 172.21.15.163:0/3178843967 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2700caad0 con 0x7fd2700b1e80 2024-07-26T06:13:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd26ea00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 0x7fd27015fe00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd260034040 tx=0x7fd2600039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.571+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26003e030 con 0x7fd2700b1e80 2024-07-26T06:13:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.572+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2600047a0 con 0x7fd2700b1e80 2024-07-26T06:13:31.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.572+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260035040 con 0x7fd2700b1e80 2024-07-26T06:13:31.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.572+0000 7fd26fe00640 1 -- 172.21.15.163:0/3178843967 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2700cadb0 con 0x7fd2700b1e80 2024-07-26T06:13:31.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.572+0000 7fd26fe00640 1 -- 172.21.15.163:0/3178843967 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2700cb370 con 0x7fd2700b1e80 2024-07-26T06:13:31.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.573+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2700cf5a0 con 0x7fd2700b1e80 2024-07-26T06:13:31.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.574+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd260041030 con 0x7fd2700b1e80 2024-07-26T06:13:31.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.576+0000 7fd266a00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd2500762d0 0x7fd250078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:31.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.576+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd2600b7b40 con 0x7fd2700b1e80 2024-07-26T06:13:31.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.576+0000 7fd26e000640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd2500762d0 0x7fd250078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:31.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.577+0000 7fd26e000640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd2500762d0 0x7fd250078790 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2700b3a80 tx=0x7fd25c007990 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:31.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.580+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd2700cf5a0 con 0x7fd2700b1e80 2024-07-26T06:13:31.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.822+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2700b57d0 con 0x7fd2700b1e80 2024-07-26T06:13:31.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.824+0000 7fd266a00640 1 -- 172.21.15.163:0/3178843967 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3918 (secure 0 0 0) 0x7fd2600817e0 con 0x7fd2700b1e80 2024-07-26T06:13:31.825 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:31.825 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":3,"btime":"2024-07-26T06:13:29:961908+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-07-26T06:13:31.825 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 3 2024-07-26T06:13:31.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.826+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd2500762d0 msgr2=0x7fd250078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.826+0000 7fd25be00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd2500762d0 0x7fd250078790 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2700b3a80 tx=0x7fd25c007990 comp rx=0 tx=0).stop 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.826+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 msgr2=0x7fd27015fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.826+0000 7fd25be00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 0x7fd27015fe00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd260034040 tx=0x7fd2600039e0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 shutdown_connections 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd2500762d0 0x7fd250078790 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd27015d5a0 0x7fd2701f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2700ce830 0x7fd2700ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 --2- 172.21.15.163:0/3178843967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2700b1e80 0x7fd27015fe00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 >> 172.21.15.163:0/3178843967 conn(0x7fd2700cfda0 msgr2=0x7fd2700b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 shutdown_connections 2024-07-26T06:13:31.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:31.827+0000 7fd25be00640 1 -- 172.21.15.163:0/3178843967 wait complete. 2024-07-26T06:13:31.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:13:32.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabccc00640 1 -- 172.21.15.163:0/4170814075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabac003a50 msgr2=0x7fabac003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabccc00640 1 --2- 172.21.15.163:0/4170814075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabac003a50 0x7fabac003e70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fabb800c290 tx=0x7fabac005830 comp rx=0 tx=0).stop 2024-07-26T06:13:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabc6800640 1 -- 172.21.15.163:0/4170814075 reap_dead start 2024-07-26T06:13:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabccc00640 1 -- 172.21.15.163:0/4170814075 shutdown_connections 2024-07-26T06:13:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabccc00640 1 -- 172.21.15.163:0/4170814075 >> 172.21.15.163:0/4170814075 conn(0x7fabc806d420 msgr2=0x7fabc806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabccc00640 1 -- 172.21.15.163:0/4170814075 shutdown_connections 2024-07-26T06:13:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.041+0000 7fabccc00640 1 -- 172.21.15.163:0/4170814075 wait complete. 2024-07-26T06:13:32.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.043+0000 7fabccc00640 1 Processor -- start 2024-07-26T06:13:32.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.044+0000 7fabccc00640 1 -- start start 2024-07-26T06:13:32.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabccc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabac003a50 0x7fabc80caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabccc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 0x7fabc81493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabccc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc8147f40 0x7fabc8157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabccc00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fabc815eea0 con 0x7fabc8146900 2024-07-26T06:13:32.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabccc00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fabc815ed20 con 0x7fabc8147f40 2024-07-26T06:13:32.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabccc00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fabc815f020 con 0x7fabac003a50 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 0x7fabc81493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 0x7fabc81493a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49368/0 (socket says 172.21.15.163:49368) 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 -- 172.21.15.163:0/1335465648 learned_addr learned my addr 172.21.15.163:0/1335465648 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabc5e00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabac003a50 0x7fabc80caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabc6800640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc8147f40 0x7fabc8157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 -- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabac003a50 msgr2=0x7fabc80caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabac003a50 0x7fabc80caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 -- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc8147f40 msgr2=0x7fabc8157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc8147f40 0x7fabc8157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabbfe00640 1 -- 172.21.15.163:0/1335465648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc81498e0 con 0x7fabc8146900 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.045+0000 7fabc5e00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabac003a50 0x7fabc80caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.046+0000 7fabbfe00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 0x7fabc81493a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fabb0010a40 tx=0x7fabb0010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.046+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb000ec80 con 0x7fabc8146900 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.046+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb000ee20 con 0x7fabc8146900 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.046+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0005280 con 0x7fabc8146900 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.046+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc8146ea0 con 0x7fabc8146900 2024-07-26T06:13:32.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.046+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc8147460 con 0x7fabc8146900 2024-07-26T06:13:32.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.047+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc806c370 con 0x7fabc8146900 2024-07-26T06:13:32.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.048+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fabb0005420 con 0x7fabc8146900 2024-07-26T06:13:32.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.049+0000 7fabbe000640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faba40762d0 0x7faba4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.049+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fabb0098130 con 0x7fabc8146900 2024-07-26T06:13:32.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.050+0000 7fabc5e00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faba40762d0 0x7faba4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.051+0000 7fabc5e00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faba40762d0 0x7faba4078790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fabb8033000 tx=0x7fabb802d000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:32.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.053+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fabc81498e0 con 0x7fabc8146900 2024-07-26T06:13:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.260+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fabc80b44b0 con 0x7fabc8146900 2024-07-26T06:13:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.261+0000 7fabbe000640 1 -- 172.21.15.163:0/1335465648 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fabb0061d40 con 0x7fabc8146900 2024-07-26T06:13:32.262 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:32.262 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:13:32.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:13:32.262 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:13:32.262 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:13:32.263 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:13:32.263 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.263+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faba40762d0 msgr2=0x7faba4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.263+0000 7fabccc00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faba40762d0 0x7faba4078790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fabb8033000 tx=0x7fabb802d000 comp rx=0 tx=0).stop 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.263+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 msgr2=0x7fabc81493a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 0x7fabc81493a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fabb0010a40 tx=0x7fabb0010f00 comp rx=0 tx=0).stop 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 shutdown_connections 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faba40762d0 0x7faba4078790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc8147f40 0x7fabc8157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabc8146900 0x7fabc81493a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 --2- 172.21.15.163:0/1335465648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabac003a50 0x7fabc80caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 >> 172.21.15.163:0/1335465648 conn(0x7fabc806d420 msgr2=0x7fabc806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 shutdown_connections 2024-07-26T06:13:32.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.264+0000 7fabccc00640 1 -- 172.21.15.163:0/1335465648 wait complete. 2024-07-26T06:13:32.283 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:13:32.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:13:32.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 -- 172.21.15.163:0/2255816470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec15d3f0 msgr2=0x7f9bec0d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 --2- 172.21.15.163:0/2255816470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec15d3f0 0x7f9bec0d2760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc005fd0 tx=0x7f9bdc02a670 comp rx=0 tx=0).stop 2024-07-26T06:13:32.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 -- 172.21.15.163:0/2255816470 shutdown_connections 2024-07-26T06:13:32.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 --2- 172.21.15.163:0/2255816470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bec0d03d0 0x7f9bec0b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 --2- 172.21.15.163:0/2255816470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec15d3f0 0x7f9bec0d2760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 -- 172.21.15.163:0/2255816470 >> 172.21.15.163:0/2255816470 conn(0x7f9bec06d880 msgr2=0x7f9bec06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:32.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 -- 172.21.15.163:0/2255816470 shutdown_connections 2024-07-26T06:13:32.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.492+0000 7f9bf2200640 1 -- 172.21.15.163:0/2255816470 wait complete. 2024-07-26T06:13:32.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.494+0000 7f9bf2200640 1 Processor -- start 2024-07-26T06:13:32.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.494+0000 7f9bf2200640 1 -- start start 2024-07-26T06:13:32.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf2200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec0d03d0 0x7f9bec1f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf2200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bec0d2ca0 0x7f9bec1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf2200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 0x7f9bec15f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf2200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec0b6dd0 con 0x7f9bec15d3f0 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf2200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec0b6c50 con 0x7f9bec0d03d0 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf2200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9bec0b6f50 con 0x7f9bec0d2ca0 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 0x7f9bec15f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 0x7f9bec15f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49376/0 (socket says 172.21.15.163:49376) 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 -- 172.21.15.163:0/773945685 learned_addr learned my addr 172.21.15.163:0/773945685 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf0e00640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec0d03d0 0x7f9bec1f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9be3e00640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bec0d2ca0 0x7f9bec1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 -- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bec0d2ca0 msgr2=0x7f9bec1534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bec0d2ca0 0x7f9bec1534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 -- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec0d03d0 msgr2=0x7f9bec1f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:32.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec0d03d0 0x7f9bec1f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:32.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.496+0000 7f9bf1800640 1 -- 172.21.15.163:0/773945685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bec0ca7b0 con 0x7f9bec15d3f0 2024-07-26T06:13:32.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.497+0000 7f9bf1800640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 0x7f9bec15f720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9be400eb40 tx=0x7f9be400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.497+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4004b20 con 0x7f9bec15d3f0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.497+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9be4004db0 con 0x7f9bec15d3f0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.497+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4005280 con 0x7f9bec15d3f0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.497+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bec0caa90 con 0x7f9bec15d3f0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.497+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bec0cb050 con 0x7f9bec15d3f0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.498+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bec0b64e0 con 0x7f9bec15d3f0 2024-07-26T06:13:32.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.499+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9be4005420 con 0x7f9bec15d3f0 2024-07-26T06:13:32.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.500+0000 7f9be9600640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9bc80762d0 0x7f9bc8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:32.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.501+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9be4098350 con 0x7f9bec15d3f0 2024-07-26T06:13:32.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.501+0000 7f9bf0e00640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9bc80762d0 0x7f9bc8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:32.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.502+0000 7f9bf0e00640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9bc80762d0 0x7f9bc8078790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc033000 tx=0x7f9bdc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:32.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.504+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9be4061f60 con 0x7f9bec15d3f0 2024-07-26T06:13:32.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:32.696+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9bec0b5490 con 0x7f9bec15d3f0 2024-07-26T06:13:33.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.004+0000 7f9be9600640 1 -- 172.21.15.163:0/773945685 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f9be4061860 con 0x7f9bec15d3f0 2024-07-26T06:13:33.006 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:13:33.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.008+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9bc80762d0 msgr2=0x7f9bc8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.008+0000 7f9bf2200640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9bc80762d0 0x7f9bc8078790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc033000 tx=0x7f9bdc004730 comp rx=0 tx=0).stop 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.008+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 msgr2=0x7f9bec15f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.008+0000 7f9bf2200640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 0x7f9bec15f720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9be400eb40 tx=0x7f9be400c2d0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.009+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 shutdown_connections 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.009+0000 7f9bf2200640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9bc80762d0 0x7f9bc8078790 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.009+0000 7f9bf2200640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bec15d3f0 0x7f9bec15f720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.009+0000 7f9bf2200640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bec0d2ca0 0x7f9bec1534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.009+0000 7f9bf2200640 1 --2- 172.21.15.163:0/773945685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bec0d03d0 0x7f9bec1f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.009+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 >> 172.21.15.163:0/773945685 conn(0x7f9bec06d880 msgr2=0x7f9bec0c9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:33.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.010+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 shutdown_connections 2024-07-26T06:13:33.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.010+0000 7f9bf2200640 1 -- 172.21.15.163:0/773945685 wait complete. 2024-07-26T06:13:33.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:13:33.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 -- 172.21.15.163:0/3015332016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00ae870 msgr2=0x7f79d00b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:33.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 --2- 172.21.15.163:0/3015332016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00ae870 0x7f79d00b3fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79d00b1a40 tx=0x7f79bc02a560 comp rx=0 tx=0).stop 2024-07-26T06:13:33.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 -- 172.21.15.163:0/3015332016 shutdown_connections 2024-07-26T06:13:33.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 --2- 172.21.15.163:0/3015332016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00ae870 0x7f79d00b3fb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 --2- 172.21.15.163:0/3015332016 >> v2:172.21.15.164:3300/0 conn(0x7f79d00b3bd0 0x7f79d00b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 -- 172.21.15.163:0/3015332016 >> 172.21.15.163:0/3015332016 conn(0x7f79d00b0e30 msgr2=0x7f79d00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:33.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 -- 172.21.15.163:0/3015332016 shutdown_connections 2024-07-26T06:13:33.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.245+0000 7f79d6800640 1 -- 172.21.15.163:0/3015332016 wait complete. 2024-07-26T06:13:33.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.247+0000 7f79d6800640 1 Processor -- start 2024-07-26T06:13:33.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.247+0000 7f79d6800640 1 -- start start 2024-07-26T06:13:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d6800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00ae870 0x7f79d0150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d6800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 0x7f79d0151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d6800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d015db10 0x7f79d01e5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d6800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79d00cb870 con 0x7f79d00b3bd0 2024-07-26T06:13:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d6800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f79d0073300 con 0x7f79d00ae870 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d6800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f79d00cb9f0 con 0x7f79d015db10 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d4a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 0x7f79d0151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d4a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 0x7f79d0151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49388/0 (socket says 172.21.15.163:49388) 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d4a00640 1 -- 172.21.15.163:0/3070550534 learned_addr learned my addr 172.21.15.163:0/3070550534 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.248+0000 7f79d5400640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00ae870 0x7f79d0150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d5e00640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d015db10 0x7f79d01e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d4a00640 1 -- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d015db10 msgr2=0x7f79d01e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d4a00640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d015db10 0x7f79d01e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d4a00640 1 -- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00ae870 msgr2=0x7f79d0150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d4a00640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00ae870 0x7f79d0150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d4a00640 1 -- 172.21.15.163:0/3070550534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d01538c0 con 0x7f79d00b3bd0 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d4a00640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 0x7f79d0151a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009850 tx=0x7f79bc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc03f070 con 0x7f79d00b3bd0 2024-07-26T06:13:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79bc0047a0 con 0x7f79d00b3bd0 2024-07-26T06:13:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc035030 con 0x7f79d00b3bd0 2024-07-26T06:13:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.249+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d0153ba0 con 0x7f79d00b3bd0 2024-07-26T06:13:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.250+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d01545c0 con 0x7f79d00b3bd0 2024-07-26T06:13:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.251+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d00b3fb0 con 0x7f79d00b3bd0 2024-07-26T06:13:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.252+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f79bc042070 con 0x7f79d00b3bd0 2024-07-26T06:13:33.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.253+0000 7f79cd600640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f79b00762d0 0x7f79b0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:33.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.253+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f79bc0b8420 con 0x7f79d00b3bd0 2024-07-26T06:13:33.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.254+0000 7f79d5400640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f79b00762d0 0x7f79b0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:33.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.255+0000 7f79d5400640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f79b00762d0 0x7f79b0078790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005fd0 tx=0x7f79c4005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:33.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.258+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f79bc081ed0 con 0x7f79d00b3bd0 2024-07-26T06:13:33.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:33.470+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f79d00b26d0 con 0x7f79d00b3bd0 2024-07-26T06:13:34.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.018+0000 7f79cd600640 1 -- 172.21.15.163:0/3070550534 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f79bc0817d0 con 0x7f79d00b3bd0 2024-07-26T06:13:34.019 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:13:34.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f79b00762d0 msgr2=0x7f79b0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:34.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f79b00762d0 0x7f79b0078790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005fd0 tx=0x7f79c4005850 comp rx=0 tx=0).stop 2024-07-26T06:13:34.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 msgr2=0x7f79d0151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:34.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 0x7f79d0151a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009850 tx=0x7f79bc003910 comp rx=0 tx=0).stop 2024-07-26T06:13:34.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 shutdown_connections 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f79b00762d0 0x7f79b0078790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d015db10 0x7f79d01e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79d00b3bd0 0x7f79d0151a40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 --2- 172.21.15.163:0/3070550534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00ae870 0x7f79d0150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 >> 172.21.15.163:0/3070550534 conn(0x7f79d00b0e30 msgr2=0x7f79d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.022+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 shutdown_connections 2024-07-26T06:13:34.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.023+0000 7f79d6800640 1 -- 172.21.15.163:0/3070550534 wait complete. 2024-07-26T06:13:34.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:13:34.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 -- 172.21.15.163:0/3995725324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18680ce350 msgr2=0x7f1868163d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:34.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 --2- 172.21.15.163:0/3995725324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18680ce350 0x7f1868163d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1860005fd0 tx=0x7f186002aa00 comp rx=0 tx=0).stop 2024-07-26T06:13:34.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 -- 172.21.15.163:0/3995725324 shutdown_connections 2024-07-26T06:13:34.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 --2- 172.21.15.163:0/3995725324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18680ce350 0x7f1868163d80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 --2- 172.21.15.163:0/3995725324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18680cd860 0x7f18680b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 -- 172.21.15.163:0/3995725324 >> 172.21.15.163:0/3995725324 conn(0x7f18680b1c40 msgr2=0x7f18680b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:34.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 -- 172.21.15.163:0/3995725324 shutdown_connections 2024-07-26T06:13:34.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.239+0000 7f186d200640 1 -- 172.21.15.163:0/3995725324 wait complete. 2024-07-26T06:13:34.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.241+0000 7f186d200640 1 Processor -- start 2024-07-26T06:13:34.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.241+0000 7f186d200640 1 -- start start 2024-07-26T06:13:34.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f186d200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 0x7f186813c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:34.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f186d200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18680cd860 0x7f18681511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f186d200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18680ce350 0x7f1868151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f186d200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1868164440 con 0x7f186806d600 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f186d200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f18681642c0 con 0x7f18680cd860 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f186d200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f18681645c0 con 0x7f18680ce350 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f1865e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 0x7f186813c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f1865e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 0x7f186813c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49416/0 (socket says 172.21.15.163:49416) 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.242+0000 7f1865e00640 1 -- 172.21.15.163:0/3242257296 learned_addr learned my addr 172.21.15.163:0/3242257296 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1866800640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18680ce350 0x7f1868151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f185fe00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18680cd860 0x7f18681511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1865e00640 1 -- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18680ce350 msgr2=0x7f1868151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1865e00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18680ce350 0x7f1868151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1865e00640 1 -- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18680cd860 msgr2=0x7f18681511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1865e00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18680cd860 0x7f18681511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:34.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1865e00640 1 -- 172.21.15.163:0/3242257296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1868158b60 con 0x7f186806d600 2024-07-26T06:13:34.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f185fe00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18680cd860 0x7f18681511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:34.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f1865e00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 0x7f186813c1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f185800d770 tx=0x7f185800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858017070 con 0x7f186806d600 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1858009d20 con 0x7f186806d600 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858005280 con 0x7f186806d600 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.243+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1868151d40 con 0x7f186806d600 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.244+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1868152300 con 0x7f186806d600 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.245+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18680b4a30 con 0x7f186806d600 2024-07-26T06:13:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.246+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1858005420 con 0x7f186806d600 2024-07-26T06:13:34.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.247+0000 7f185e000640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18440762d0 0x7f1844078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.247+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f18580978e0 con 0x7f186806d600 2024-07-26T06:13:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.247+0000 7f185fe00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18440762d0 0x7f1844078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.249+0000 7f185fe00640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18440762d0 0x7f1844078790 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f18681f7670 tx=0x7f1850009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:34.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.252+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1858061200 con 0x7f186806d600 2024-07-26T06:13:34.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:34.508+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1868156cd0 con 0x7f186806d600 2024-07-26T06:13:35.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.022+0000 7f185e000640 1 -- 172.21.15.163:0/3242257296 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f18580691f0 con 0x7f186806d600 2024-07-26T06:13:35.024 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T06:13:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18440762d0 msgr2=0x7f1844078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18440762d0 0x7f1844078790 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f18681f7670 tx=0x7f1850009040 comp rx=0 tx=0).stop 2024-07-26T06:13:35.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 msgr2=0x7f186813c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 0x7f186813c1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f185800d770 tx=0x7f185800dc30 comp rx=0 tx=0).stop 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 shutdown_connections 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18440762d0 0x7f1844078790 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18680ce350 0x7f1868151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18680cd860 0x7f18681511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 --2- 172.21.15.163:0/3242257296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f186806d600 0x7f186813c1f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.026+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 >> 172.21.15.163:0/3242257296 conn(0x7f18680b1c40 msgr2=0x7f1868163af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.027+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 shutdown_connections 2024-07-26T06:13:35.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.027+0000 7f186d200640 1 -- 172.21.15.163:0/3242257296 wait complete. 2024-07-26T06:13:35.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:35.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.245+0000 7f1b13e00640 1 -- 172.21.15.163:0/2221474859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140cc1c0 msgr2=0x7f1b140cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.245+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2221474859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140cc1c0 0x7f1b140cd0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b14071230 tx=0x7f1b0002a560 comp rx=0 tx=0).stop 2024-07-26T06:13:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.246+0000 7f1b13e00640 1 -- 172.21.15.163:0/2221474859 shutdown_connections 2024-07-26T06:13:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.246+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2221474859 >> v2:172.21.15.164:3300/0 conn(0x7f1b1415a6b0 0x7f1b140b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.246+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2221474859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140cc1c0 0x7f1b140cd0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.246+0000 7f1b13e00640 1 -- 172.21.15.163:0/2221474859 >> 172.21.15.163:0/2221474859 conn(0x7f1b140b0e30 msgr2=0x7f1b140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.246+0000 7f1b13e00640 1 -- 172.21.15.163:0/2221474859 shutdown_connections 2024-07-26T06:13:35.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.246+0000 7f1b13e00640 1 -- 172.21.15.163:0/2221474859 wait complete. 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.247+0000 7f1b13e00640 1 Processor -- start 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.247+0000 7f1b13e00640 1 -- start start 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b13e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 0x7f1b14148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b13e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b140cc1c0 0x7f1b14149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b13e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b1415a6b0 0x7f1b141f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b13e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1b140b25c0 con 0x7f1b140741f0 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b13e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1b140b2440 con 0x7f1b1415a6b0 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b13e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1b140b2740 con 0x7f1b140cc1c0 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b12a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 0x7f1b14148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.248+0000 7f1b12a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 0x7f1b14148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44714/0 (socket says 172.21.15.163:44714) 2024-07-26T06:13:35.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 -- 172.21.15.163:0/2681737157 learned_addr learned my addr 172.21.15.163:0/2681737157 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b13400640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b1415a6b0 0x7f1b141f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 -- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b140cc1c0 msgr2=0x7f1b14149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b140cc1c0 0x7f1b14149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 -- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b1415a6b0 msgr2=0x7f1b141f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b1415a6b0 0x7f1b141f5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 -- 172.21.15.163:0/2681737157 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1415ae30 con 0x7f1b140741f0 2024-07-26T06:13:35.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b12a00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 0x7f1b14148210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400b420 tx=0x7f1b0400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:35.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04011830 con 0x7f1b140741f0 2024-07-26T06:13:35.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b04016070 con 0x7f1b140741f0 2024-07-26T06:13:35.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04011d60 con 0x7f1b140741f0 2024-07-26T06:13:35.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.249+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b14150050 con 0x7f1b140741f0 2024-07-26T06:13:35.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.250+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b14150610 con 0x7f1b140741f0 2024-07-26T06:13:35.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.251+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b140cd0c0 con 0x7f1b140741f0 2024-07-26T06:13:35.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.252+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1b0401b460 con 0x7f1b140741f0 2024-07-26T06:13:35.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.253+0000 7f1b0aa00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1af40762d0 0x7f1af4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.253+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1b040972d0 con 0x7f1b140741f0 2024-07-26T06:13:35.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.253+0000 7f1b12000640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1af40762d0 0x7f1af4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.254+0000 7f1b12000640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1af40762d0 0x7f1af4078790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b00033000 tx=0x7f1b000046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:35.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.257+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1b04020020 con 0x7f1b140741f0 2024-07-26T06:13:35.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.448+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b140b4aa0 con 0x7f1b140741f0 2024-07-26T06:13:35.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.451+0000 7f1b0aa00640 1 -- 172.21.15.163:0/2681737157 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16642 (secure 0 0 0) 0x7f1b04060c20 con 0x7f1b140741f0 2024-07-26T06:13:35.452 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:35.452 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":20,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:34.510713+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1af40762d0 msgr2=0x7f1af4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1af40762d0 0x7f1af4078790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b00033000 tx=0x7f1b000046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 msgr2=0x7f1b14148210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 0x7f1b14148210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400b420 tx=0x7f1b0400b8e0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 shutdown_connections 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1af40762d0 0x7f1af4078790 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b1415a6b0 0x7f1b141f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b140cc1c0 0x7f1b14149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.453+0000 7f1b13e00640 1 --2- 172.21.15.163:0/2681737157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b140741f0 0x7f1b14148210 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.454+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 >> 172.21.15.163:0/2681737157 conn(0x7f1b140b0e30 msgr2=0x7f1b14157850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.454+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 shutdown_connections 2024-07-26T06:13:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.454+0000 7f1b13e00640 1 -- 172.21.15.163:0/2681737157 wait complete. 2024-07-26T06:13:35.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:35.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 -- 172.21.15.163:0/4247641179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e00b3bd0 msgr2=0x7f40e00b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 --2- 172.21.15.163:0/4247641179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e00b3bd0 0x7f40e00b4bd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009830 tx=0x7f40d002a480 comp rx=0 tx=0).stop 2024-07-26T06:13:35.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 -- 172.21.15.163:0/4247641179 shutdown_connections 2024-07-26T06:13:35.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 --2- 172.21.15.163:0/4247641179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e00b3bd0 0x7f40e00b4bd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 -- 172.21.15.163:0/4247641179 >> 172.21.15.163:0/4247641179 conn(0x7f40e00b0e30 msgr2=0x7f40e00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:35.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 -- 172.21.15.163:0/4247641179 shutdown_connections 2024-07-26T06:13:35.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.674+0000 7f40e6400640 1 -- 172.21.15.163:0/4247641179 wait complete. 2024-07-26T06:13:35.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.676+0000 7f40e6400640 1 Processor -- start 2024-07-26T06:13:35.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.676+0000 7f40e6400640 1 -- start start 2024-07-26T06:13:35.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e6400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 0x7f40e0148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e6400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e00ae870 0x7f40e01594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e6400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40e00b3bd0 0x7f40e0157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e6400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40e015a1d0 con 0x7f40e0073820 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e6400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40e015a350 con 0x7f40e00ae870 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e6400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f40e00743c0 con 0x7f40e00b3bd0 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 0x7f40e0148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 0x7f40e0148b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44732/0 (socket says 172.21.15.163:44732) 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 -- 172.21.15.163:0/1202423221 learned_addr learned my addr 172.21.15.163:0/1202423221 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5a00640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40e00b3bd0 0x7f40e0157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40d7e00640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e00ae870 0x7f40e01594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 -- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40e00b3bd0 msgr2=0x7f40e0157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40e00b3bd0 0x7f40e0157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 -- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e00ae870 msgr2=0x7f40e01594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e00ae870 0x7f40e01594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40e5000640 1 -- 172.21.15.163:0/1202423221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e0159a30 con 0x7f40e0073820 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.677+0000 7f40d7e00640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e00ae870 0x7f40e01594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.678+0000 7f40e5000640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 0x7f40e0148b90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009950 tx=0x7f40d0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.678+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0031d60 con 0x7f40e0073820 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.678+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40d0033e10 con 0x7f40e0073820 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.678+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e0158450 con 0x7f40e0073820 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.678+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d00332b0 con 0x7f40e0073820 2024-07-26T06:13:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.678+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e0158a10 con 0x7f40e0073820 2024-07-26T06:13:35.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.679+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e006fb10 con 0x7f40e0073820 2024-07-26T06:13:35.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.680+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f40d0033450 con 0x7f40e0073820 2024-07-26T06:13:35.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.682+0000 7f40dd600640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40bc0762d0 0x7f40bc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:35.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.682+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f40d00b8a90 con 0x7f40e0073820 2024-07-26T06:13:35.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.682+0000 7f40d7e00640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40bc0762d0 0x7f40bc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:35.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.683+0000 7f40d7e00640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40bc0762d0 0x7f40bc078790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f40c8001220 tx=0x7f40c8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:35.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.686+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f40e006fb10 con 0x7f40e0073820 2024-07-26T06:13:35.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.896+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40e00b26d0 con 0x7f40e0073820 2024-07-26T06:13:35.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.899+0000 7f40dd600640 1 -- 172.21.15.163:0/1202423221 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16642 (secure 0 0 0) 0x7f40e0159a30 con 0x7f40e0073820 2024-07-26T06:13:35.900 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:35.900 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":20,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:34.510713+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40bc0762d0 msgr2=0x7f40bc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40bc0762d0 0x7f40bc078790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f40c8001220 tx=0x7f40c8009040 comp rx=0 tx=0).stop 2024-07-26T06:13:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 msgr2=0x7f40e0148b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 0x7f40e0148b90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009950 tx=0x7f40d0033c20 comp rx=0 tx=0).stop 2024-07-26T06:13:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 shutdown_connections 2024-07-26T06:13:35.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40bc0762d0 0x7f40bc078790 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40e00b3bd0 0x7f40e0157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e00ae870 0x7f40e01594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 --2- 172.21.15.163:0/1202423221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e0073820 0x7f40e0148b90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 >> 172.21.15.163:0/1202423221 conn(0x7f40e00b0e30 msgr2=0x7f40e00aeff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 shutdown_connections 2024-07-26T06:13:35.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:35.902+0000 7f40e6400640 1 -- 172.21.15.163:0/1202423221 wait complete. 2024-07-26T06:13:35.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:13:36.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f1145000640 1 -- 172.21.15.163:0/2482505081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1120004d40 msgr2=0x7f11200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:36.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f1145000640 1 --2- 172.21.15.163:0/2482505081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1120004d40 0x7f11200051c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1138005fd0 tx=0x7f1120005830 comp rx=0 tx=0).stop 2024-07-26T06:13:36.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f113e800640 1 -- 172.21.15.163:0/2482505081 reap_dead start 2024-07-26T06:13:36.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f1145000640 1 -- 172.21.15.163:0/2482505081 shutdown_connections 2024-07-26T06:13:36.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f1145000640 1 -- 172.21.15.163:0/2482505081 >> 172.21.15.163:0/2482505081 conn(0x7f114006bf40 msgr2=0x7f114006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:36.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f1145000640 1 -- 172.21.15.163:0/2482505081 shutdown_connections 2024-07-26T06:13:36.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.128+0000 7f1145000640 1 -- 172.21.15.163:0/2482505081 wait complete. 2024-07-26T06:13:36.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.130+0000 7f1145000640 1 Processor -- start 2024-07-26T06:13:36.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.130+0000 7f1145000640 1 -- start start 2024-07-26T06:13:36.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1145000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11401f1040 0x7f11400cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:36.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1145000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11400cb570 0x7f1140150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:36.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1145000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 0x7f114014fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:36.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1145000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1140073ad0 con 0x7f11401511e0 2024-07-26T06:13:36.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1145000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1140073950 con 0x7f11401f1040 2024-07-26T06:13:36.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1145000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1140073c50 con 0x7f11400cb570 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f113e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 0x7f114014fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f113e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 0x7f114014fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44760/0 (socket says 172.21.15.163:44760) 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f113e800640 1 -- 172.21.15.163:0/2294804066 learned_addr learned my addr 172.21.15.163:0/2294804066 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f113de00640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11401f1040 0x7f11400cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.131+0000 7f1137e00640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11400cb570 0x7f1140150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113e800640 1 -- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11400cb570 msgr2=0x7f1140150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113e800640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11400cb570 0x7f1140150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113e800640 1 -- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11401f1040 msgr2=0x7f11400cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113e800640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11401f1040 0x7f11400cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113e800640 1 -- 172.21.15.163:0/2294804066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11401f3d50 con 0x7f11401511e0 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113de00640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11401f1040 0x7f11400cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f113e800640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 0x7f114014fbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1138005fd0 tx=0x7f1138033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:36.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1138038040 con 0x7f11401511e0 2024-07-26T06:13:36.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1138004b20 con 0x7f11401511e0 2024-07-26T06:13:36.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1138003970 con 0x7f11401511e0 2024-07-26T06:13:36.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11401f4030 con 0x7f11401511e0 2024-07-26T06:13:36.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.132+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11401f45f0 con 0x7f11401511e0 2024-07-26T06:13:36.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.133+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1140069810 con 0x7f11401511e0 2024-07-26T06:13:36.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.135+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1138003b10 con 0x7f11401511e0 2024-07-26T06:13:36.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.135+0000 7f1136000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f111c0762d0 0x7f111c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:36.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.136+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f11380b73c0 con 0x7f11401511e0 2024-07-26T06:13:36.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.136+0000 7f113de00640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f111c0762d0 0x7f111c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:36.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.137+0000 7f113de00640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f111c0762d0 0x7f111c078790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1128005fd0 tx=0x7f1128005d10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:36.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.140+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1138080db0 con 0x7f11401511e0 2024-07-26T06:13:36.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:36.369+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f11400cc510 con 0x7f11401511e0 2024-07-26T06:13:37.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.066+0000 7f1136000640 1 -- 172.21.15.163:0/2294804066 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f11380806b0 con 0x7f11401511e0 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.069+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f111c0762d0 msgr2=0x7f111c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.069+0000 7f1145000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f111c0762d0 0x7f111c078790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1128005fd0 tx=0x7f1128005d10 comp rx=0 tx=0).stop 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.069+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 msgr2=0x7f114014fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.069+0000 7f1145000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 0x7f114014fbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1138005fd0 tx=0x7f1138033670 comp rx=0 tx=0).stop 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 shutdown_connections 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f111c0762d0 0x7f111c078790 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11401511e0 0x7f114014fbe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11400cb570 0x7f1140150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 --2- 172.21.15.163:0/2294804066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11401f1040 0x7f11400cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 >> 172.21.15.163:0/2294804066 conn(0x7f114006bf40 msgr2=0x7f11400cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:37.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 shutdown_connections 2024-07-26T06:13:37.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.070+0000 7f1145000640 1 -- 172.21.15.163:0/2294804066 wait complete. 2024-07-26T06:13:37.091 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:13:37.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:13:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 -- 172.21.15.163:0/1047896102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc15f060 msgr2=0x7f1cbc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 --2- 172.21.15.163:0/1047896102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc15f060 0x7f1cbc06d180 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0749d0 tx=0x7f1cb402a820 comp rx=0 tx=0).stop 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 -- 172.21.15.163:0/1047896102 shutdown_connections 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 --2- 172.21.15.163:0/1047896102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc15f060 0x7f1cbc06d180 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 --2- 172.21.15.163:0/1047896102 >> v2:172.21.15.164:3300/0 conn(0x7f1cbc0aebd0 0x7f1cbc15eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 -- 172.21.15.163:0/1047896102 >> 172.21.15.163:0/1047896102 conn(0x7f1cbc0b0510 msgr2=0x7f1cbc0b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 -- 172.21.15.163:0/1047896102 shutdown_connections 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.292+0000 7f1cc3000640 1 -- 172.21.15.163:0/1047896102 wait complete. 2024-07-26T06:13:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.294+0000 7f1cc3000640 1 Processor -- start 2024-07-26T06:13:37.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.294+0000 7f1cc3000640 1 -- start start 2024-07-26T06:13:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc3000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 0x7f1cbc14ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc3000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cbc152090 0x7f1cbc156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc3000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cbc15f060 0x7f1cbc1549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc3000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc15f990 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc3000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc15f810 con 0x7f1cbc15f060 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc3000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc06d6c0 con 0x7f1cbc152090 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc1c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 0x7f1cbc14ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc1c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 0x7f1cbc14ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44784/0 (socket says 172.21.15.163:44784) 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.295+0000 7f1cc1c00640 1 -- 172.21.15.163:0/3198445286 learned_addr learned my addr 172.21.15.163:0/3198445286 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc1c00640 1 -- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cbc152090 msgr2=0x7f1cbc156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc1c00640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cbc152090 0x7f1cbc156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc2600640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cbc15f060 0x7f1cbc1549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc1c00640 1 -- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cbc15f060 msgr2=0x7f1cbc1549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc1c00640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cbc15f060 0x7f1cbc1549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc1c00640 1 -- 172.21.15.163:0/3198445286 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc1f4e00 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc1c00640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 0x7f1cbc14ff80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000c730 tx=0x7f1cb000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0006a90 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1cb0006c30 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0007030 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cbc1f50e0 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.296+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cbc156cd0 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.297+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cbc06db20 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.298+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1cb00071d0 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.300+0000 7f1cb9600640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c9c0762d0 0x7f1c9c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:37.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.300+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1cb00973c0 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.301+0000 7f1cc1200640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c9c0762d0 0x7f1c9c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:37.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.302+0000 7f1cc1200640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c9c0762d0 0x7f1c9c078790 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8005fd0 tx=0x7f1ca8005c70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.304+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cbc15f470 con 0x7f1cbc0aebd0 2024-07-26T06:13:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:37.548+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1cbc0af2b0 con 0x7f1cbc0aebd0 2024-07-26T06:13:38.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.070+0000 7f1cb9600640 1 -- 172.21.15.163:0/3198445286 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f1cb0060c80 con 0x7f1cbc0aebd0 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c9c0762d0 msgr2=0x7f1c9c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c9c0762d0 0x7f1c9c078790 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8005fd0 tx=0x7f1ca8005c70 comp rx=0 tx=0).stop 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 msgr2=0x7f1cbc14ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 0x7f1cbc14ff80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000c730 tx=0x7f1cb000cbf0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 shutdown_connections 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c9c0762d0 0x7f1c9c078790 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cbc15f060 0x7f1cbc1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cbc152090 0x7f1cbc156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 --2- 172.21.15.163:0/3198445286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cbc0aebd0 0x7f1cbc14ff80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 >> 172.21.15.163:0/3198445286 conn(0x7f1cbc0b0510 msgr2=0x7f1cbc06f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:38.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 shutdown_connections 2024-07-26T06:13:38.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.073+0000 7f1cc3000640 1 -- 172.21.15.163:0/3198445286 wait complete. 2024-07-26T06:13:38.095 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:13:38.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:13:38.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 -- 172.21.15.163:0/1579692490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 msgr2=0x7f9c300b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/1579692490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c300b6bd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c300690b0 tx=0x7f9c2802a560 comp rx=0 tx=0).stop 2024-07-26T06:13:38.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 -- 172.21.15.163:0/1579692490 shutdown_connections 2024-07-26T06:13:38.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/1579692490 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c300b6bd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/1579692490 >> v2:172.21.15.164:3300/0 conn(0x7f9c3015d5a0 0x7f9c300ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 -- 172.21.15.163:0/1579692490 >> 172.21.15.163:0/1579692490 conn(0x7f9c300cfda0 msgr2=0x7f9c300d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:38.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 -- 172.21.15.163:0/1579692490 shutdown_connections 2024-07-26T06:13:38.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.301+0000 7f9c2fe00640 1 -- 172.21.15.163:0/1579692490 wait complete. 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.303+0000 7f9c2fe00640 1 Processor -- start 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.303+0000 7f9c2fe00640 1 -- start start 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c3014a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c300ce830 0x7f9c3015d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c3015d5a0 0x7f9c301575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300b5130 con 0x7f9c300b1e80 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300b4fb0 con 0x7f9c3015d5a0 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9c300b52b0 con 0x7f9c300ce830 2024-07-26T06:13:38.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c3014a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c3014a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44800/0 (socket says 172.21.15.163:44800) 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 -- 172.21.15.163:0/135604182 learned_addr learned my addr 172.21.15.163:0/135604182 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 -- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c300ce830 msgr2=0x7f9c3015d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c300ce830 0x7f9c3015d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 -- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c3015d5a0 msgr2=0x7f9c301575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c3015d5a0 0x7f9c301575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.304+0000 7f9c2ea00640 1 -- 172.21.15.163:0/135604182 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c30156540 con 0x7f9c300b1e80 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.305+0000 7f9c2ea00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c3014a830 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000ea50 tx=0x7f9c2000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.305+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2000cdc0 con 0x7f9c300b1e80 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.305+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c20011560 con 0x7f9c300b1e80 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.305+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c30156820 con 0x7f9c300b1e80 2024-07-26T06:13:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.305+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20011990 con 0x7f9c300b1e80 2024-07-26T06:13:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.305+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c30156de0 con 0x7f9c300b1e80 2024-07-26T06:13:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.306+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c300b3b80 con 0x7f9c300b1e80 2024-07-26T06:13:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.307+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9c20011b30 con 0x7f9c300b1e80 2024-07-26T06:13:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.309+0000 7f9c26a00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9c100762d0 0x7f9c10078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.309+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9c200983a0 con 0x7f9c300b1e80 2024-07-26T06:13:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.309+0000 7f9c2e000640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9c100762d0 0x7f9c10078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:38.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.311+0000 7f9c2e000640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9c100762d0 0x7f9c10078790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c000f60 tx=0x7f9c1c0078b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.313+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9c300b3b80 con 0x7f9c300b1e80 2024-07-26T06:13:38.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.562+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c30074680 con 0x7f9c300b1e80 2024-07-26T06:13:38.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.563+0000 7f9c26a00640 1 -- 172.21.15.163:0/135604182 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7f9c20016030 con 0x7f9c300b1e80 2024-07-26T06:13:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:38.564 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:13:38.565 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:13:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9c100762d0 msgr2=0x7f9c10078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9c100762d0 0x7f9c10078790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c000f60 tx=0x7f9c1c0078b0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 msgr2=0x7f9c3014a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c3014a830 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000ea50 tx=0x7f9c2000ef10 comp rx=0 tx=0).stop 2024-07-26T06:13:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 shutdown_connections 2024-07-26T06:13:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9c100762d0 0x7f9c10078790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c3015d5a0 0x7f9c301575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c300ce830 0x7f9c3015d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 --2- 172.21.15.163:0/135604182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c300b1e80 0x7f9c3014a830 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 >> 172.21.15.163:0/135604182 conn(0x7f9c300cfda0 msgr2=0x7f9c300b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 shutdown_connections 2024-07-26T06:13:38.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.566+0000 7f9c2fe00640 1 -- 172.21.15.163:0/135604182 wait complete. 2024-07-26T06:13:38.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5877000640 1 -- 172.21.15.163:0/3165258657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5858003a80 msgr2=0x7f5858003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5877000640 1 --2- 172.21.15.163:0/3165258657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5858003a80 0x7f5858003ea0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f58600016c0 tx=0x7f5858005830 comp rx=0 tx=0).stop 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5876600640 1 -- 172.21.15.163:0/3165258657 reap_dead start 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5877000640 1 -- 172.21.15.163:0/3165258657 shutdown_connections 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5877000640 1 -- 172.21.15.163:0/3165258657 >> 172.21.15.163:0/3165258657 conn(0x7f58700cfda0 msgr2=0x7f58700d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5877000640 1 -- 172.21.15.163:0/3165258657 shutdown_connections 2024-07-26T06:13:38.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.782+0000 7f5877000640 1 -- 172.21.15.163:0/3165258657 wait complete. 2024-07-26T06:13:38.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.784+0000 7f5877000640 1 Processor -- start 2024-07-26T06:13:38.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.784+0000 7f5877000640 1 -- start start 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.784+0000 7f5877000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58700b38e0 0x7f58701555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.784+0000 7f5877000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58700b3cc0 0x7f58700cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5877000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 0x7f587014c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5877000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f58700b6f60 con 0x7f587014c450 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5877000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f58700b6de0 con 0x7f58700b38e0 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5877000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f58700b4da0 con 0x7f58700b3cc0 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 0x7f587014c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 0x7f587014c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44820/0 (socket says 172.21.15.163:44820) 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 -- 172.21.15.163:0/137151009 learned_addr learned my addr 172.21.15.163:0/137151009 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:38.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5875c00640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58700b38e0 0x7f58701555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5875200640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58700b3cc0 0x7f58700cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 -- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58700b3cc0 msgr2=0x7f58700cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58700b3cc0 0x7f58700cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 -- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58700b38e0 msgr2=0x7f58701555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58700b38e0 0x7f58701555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5876600640 1 -- 172.21.15.163:0/137151009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5870155b30 con 0x7f587014c450 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.785+0000 7f5875c00640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58700b38e0 0x7f58701555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.786+0000 7f5876600640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 0x7f587014c8e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f587008c7d0 tx=0x7f586800fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.786+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5868005220 con 0x7f587014c450 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.786+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58680054b0 con 0x7f587014c450 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.786+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58700cd6b0 con 0x7f587014c450 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.786+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5868004a00 con 0x7f587014c450 2024-07-26T06:13:38.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.786+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58700cc3b0 con 0x7f587014c450 2024-07-26T06:13:38.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.787+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58700b64c0 con 0x7f587014c450 2024-07-26T06:13:38.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.788+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5868019030 con 0x7f587014c450 2024-07-26T06:13:38.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.789+0000 7f586d600640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f58540762d0 0x7f5854078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:38.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.789+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5868097930 con 0x7f587014c450 2024-07-26T06:13:38.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.790+0000 7f5875c00640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f58540762d0 0x7f5854078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:38.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.791+0000 7f5875c00640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f58540762d0 0x7f5854078790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5860032000 tx=0x7f5860001190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:38.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:38.793+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5868060c30 con 0x7f587014c450 2024-07-26T06:13:39.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.002+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58700ce610 con 0x7f587014c450 2024-07-26T06:13:39.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.005+0000 7f586d600640 1 -- 172.21.15.163:0/137151009 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7f5868069270 con 0x7f587014c450 2024-07-26T06:13:39.006 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:39.007 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":21,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:36.040341+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:39.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f58540762d0 msgr2=0x7f5854078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f58540762d0 0x7f5854078790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5860032000 tx=0x7f5860001190 comp rx=0 tx=0).stop 2024-07-26T06:13:39.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 msgr2=0x7f587014c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 0x7f587014c8e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f587008c7d0 tx=0x7f586800fa20 comp rx=0 tx=0).stop 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 shutdown_connections 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f58540762d0 0x7f5854078790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f587014c450 0x7f587014c8e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58700b3cc0 0x7f58700cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 --2- 172.21.15.163:0/137151009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58700b38e0 0x7f58701555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 >> 172.21.15.163:0/137151009 conn(0x7f58700cfda0 msgr2=0x7f58700d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:39.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 shutdown_connections 2024-07-26T06:13:39.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.008+0000 7f5877000640 1 -- 172.21.15.163:0/137151009 wait complete. 2024-07-26T06:13:39.028 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:13:39.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:13:39.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc73200640 1 -- 172.21.15.163:0/1181659941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 msgr2=0x7fdc580062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc73200640 1 --2- 172.21.15.163:0/1181659941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc580062d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009f00 tx=0x7fdc58006940 comp rx=0 tx=0).stop 2024-07-26T06:13:39.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc72800640 1 -- 172.21.15.163:0/1181659941 reap_dead start 2024-07-26T06:13:39.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc73200640 1 -- 172.21.15.163:0/1181659941 shutdown_connections 2024-07-26T06:13:39.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc73200640 1 -- 172.21.15.163:0/1181659941 >> 172.21.15.163:0/1181659941 conn(0x7fdc740b0e30 msgr2=0x7fdc740b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:39.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc73200640 1 -- 172.21.15.163:0/1181659941 shutdown_connections 2024-07-26T06:13:39.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.218+0000 7fdc73200640 1 -- 172.21.15.163:0/1181659941 wait complete. 2024-07-26T06:13:39.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.220+0000 7fdc73200640 1 Processor -- start 2024-07-26T06:13:39.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.220+0000 7fdc73200640 1 -- start start 2024-07-26T06:13:39.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc73200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc741551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc73200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc741502e0 0x7fdc74158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc73200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc741592f0 0x7fdc7415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc73200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc740cd240 con 0x7fdc58005e50 2024-07-26T06:13:39.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc73200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc740cd0c0 con 0x7fdc741592f0 2024-07-26T06:13:39.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc73200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdc740cd3c0 con 0x7fdc741502e0 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc71e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc741551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc71e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc741551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44838/0 (socket says 172.21.15.163:44838) 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc71e00640 1 -- 172.21.15.163:0/4151211123 learned_addr learned my addr 172.21.15.163:0/4151211123 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc71e00640 1 -- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc741502e0 msgr2=0x7fdc74158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.221+0000 7fdc6be00640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc741502e0 0x7fdc74158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc71e00640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc741502e0 0x7fdc74158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc71e00640 1 -- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc741592f0 msgr2=0x7fdc7415ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc71e00640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc741592f0 0x7fdc7415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc71e00640 1 -- 172.21.15.163:0/4151211123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc741556e0 con 0x7fdc58005e50 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc71e00640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc741551a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400e7a0 tx=0x7fdc6400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400cbe0 con 0x7fdc58005e50 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc6400ce70 con 0x7fdc58005e50 2024-07-26T06:13:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64005260 con 0x7fdc58005e50 2024-07-26T06:13:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc74150ea0 con 0x7fdc58005e50 2024-07-26T06:13:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.222+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc74151460 con 0x7fdc58005e50 2024-07-26T06:13:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.223+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc740cbab0 con 0x7fdc58005e50 2024-07-26T06:13:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.224+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc64005400 con 0x7fdc58005e50 2024-07-26T06:13:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.226+0000 7fdc6a000640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc500762d0 0x7fdc50078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.226+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdc64097e70 con 0x7fdc58005e50 2024-07-26T06:13:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.226+0000 7fdc6be00640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc500762d0 0x7fdc50078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.227+0000 7fdc6be00640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc500762d0 0x7fdc50078790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c000f60 tx=0x7fdc5c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:39.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.230+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc64016030 con 0x7fdc58005e50 2024-07-26T06:13:39.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.417+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc74159710 con 0x7fdc500762d0 2024-07-26T06:13:39.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.432+0000 7fdc6a000640 1 -- 172.21.15.163:0/4151211123 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251929 (secure 0 0 0) 0x7fdc74159710 con 0x7fdc500762d0 2024-07-26T06:13:39.438 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:39.439 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc500762d0 msgr2=0x7fdc50078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc500762d0 0x7fdc50078790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c000f60 tx=0x7fdc5c009040 comp rx=0 tx=0).stop 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 msgr2=0x7fdc741551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc741551a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400e7a0 tx=0x7fdc6400ec60 comp rx=0 tx=0).stop 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 shutdown_connections 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc500762d0 0x7fdc50078790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc741592f0 0x7fdc7415ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc741502e0 0x7fdc74158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 --2- 172.21.15.163:0/4151211123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc58005e50 0x7fdc741551a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 >> 172.21.15.163:0/4151211123 conn(0x7fdc740b0e30 msgr2=0x7fdc740b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:39.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 shutdown_connections 2024-07-26T06:13:39.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.439+0000 7fdc73200640 1 -- 172.21.15.163:0/4151211123 wait complete. 2024-07-26T06:13:39.458 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-07-26T06:13:37.644682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2875392,"data_stored":1740684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.637064"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267640+0000","last_change":"2024-07-26T06:13:35.072083+0000","last_active":"2024-07-26T06:13:37.267640+0000","last_peered":"2024-07-26T06:13:37.267640+0000","last_clean":"2024-07-26T06:13:37.267640+0000","last_became_active":"2024-07-26T06:13:35.071746+0000","last_became_peered":"2024-07-26T06:13:35.071746+0000","last_unstale":"2024-07-26T06:13:37.267640+0000","last_undegraded":"2024-07-26T06:13:37.267640+0000","last_fullsized":"2024-07-26T06:13:37.267640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.063250+0000","last_change":"2024-07-26T06:13:35.058738+0000","last_active":"2024-07-26T06:13:35.063250+0000","last_peered":"2024-07-26T06:13:35.063250+0000","last_clean":"2024-07-26T06:13:35.063250+0000","last_became_active":"2024-07-26T06:13:35.058470+0000","last_became_peered":"2024-07-26T06:13:35.058470+0000","last_unstale":"2024-07-26T06:13:35.063250+0000","last_undegraded":"2024-07-26T06:13:35.063250+0000","last_fullsized":"2024-07-26T06:13:35.063250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.331691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056891+0000","last_change":"2024-07-26T06:13:35.042867+0000","last_active":"2024-07-26T06:13:36.056891+0000","last_peered":"2024-07-26T06:13:36.056891+0000","last_clean":"2024-07-26T06:13:36.056891+0000","last_became_active":"2024-07-26T06:13:35.042660+0000","last_became_peered":"2024-07-26T06:13:35.042660+0000","last_unstale":"2024-07-26T06:13:36.056891+0000","last_undegraded":"2024-07-26T06:13:36.056891+0000","last_fullsized":"2024-07-26T06:13:36.056891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:17:02.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165050+0000","last_change":"2024-07-26T06:13:35.076112+0000","last_active":"2024-07-26T06:13:36.165050+0000","last_peered":"2024-07-26T06:13:36.165050+0000","last_clean":"2024-07-26T06:13:36.165050+0000","last_became_active":"2024-07-26T06:13:35.075903+0000","last_became_peered":"2024-07-26T06:13:35.075903+0000","last_unstale":"2024-07-26T06:13:36.165050+0000","last_undegraded":"2024-07-26T06:13:36.165050+0000","last_fullsized":"2024-07-26T06:13:36.165050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267271+0000","last_change":"2024-07-26T06:13:35.068273+0000","last_active":"2024-07-26T06:13:37.267271+0000","last_peered":"2024-07-26T06:13:37.267271+0000","last_clean":"2024-07-26T06:13:37.267271+0000","last_became_active":"2024-07-26T06:13:35.067645+0000","last_became_peered":"2024-07-26T06:13:35.067645+0000","last_unstale":"2024-07-26T06:13:37.267271+0000","last_undegraded":"2024-07-26T06:13:37.267271+0000","last_fullsized":"2024-07-26T06:13:37.267271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164727+0000","last_change":"2024-07-26T06:13:35.080356+0000","last_active":"2024-07-26T06:13:36.164727+0000","last_peered":"2024-07-26T06:13:36.164727+0000","last_clean":"2024-07-26T06:13:36.164727+0000","last_became_active":"2024-07-26T06:13:35.079906+0000","last_became_peered":"2024-07-26T06:13:35.079906+0000","last_unstale":"2024-07-26T06:13:36.164727+0000","last_undegraded":"2024-07-26T06:13:36.164727+0000","last_fullsized":"2024-07-26T06:13:36.164727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266764+0000","last_change":"2024-07-26T06:13:35.070227+0000","last_active":"2024-07-26T06:13:37.266764+0000","last_peered":"2024-07-26T06:13:37.266764+0000","last_clean":"2024-07-26T06:13:37.266764+0000","last_became_active":"2024-07-26T06:13:35.069602+0000","last_became_peered":"2024-07-26T06:13:35.069602+0000","last_unstale":"2024-07-26T06:13:37.266764+0000","last_undegraded":"2024-07-26T06:13:37.266764+0000","last_fullsized":"2024-07-26T06:13:37.266764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059249+0000","last_change":"2024-07-26T06:13:35.077098+0000","last_active":"2024-07-26T06:13:36.059249+0000","last_peered":"2024-07-26T06:13:36.059249+0000","last_clean":"2024-07-26T06:13:36.059249+0000","last_became_active":"2024-07-26T06:13:35.076814+0000","last_became_peered":"2024-07-26T06:13:35.076814+0000","last_unstale":"2024-07-26T06:13:36.059249+0000","last_undegraded":"2024-07-26T06:13:36.059249+0000","last_fullsized":"2024-07-26T06:13:36.059249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:19.325249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267856+0000","last_change":"2024-07-26T06:13:35.081415+0000","last_active":"2024-07-26T06:13:37.267856+0000","last_peered":"2024-07-26T06:13:37.267856+0000","last_clean":"2024-07-26T06:13:37.267856+0000","last_became_active":"2024-07-26T06:13:35.080563+0000","last_became_peered":"2024-07-26T06:13:35.080563+0000","last_unstale":"2024-07-26T06:13:37.267856+0000","last_undegraded":"2024-07-26T06:13:37.267856+0000","last_fullsized":"2024-07-26T06:13:37.267856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241645+0000","last_change":"2024-07-26T06:13:35.067607+0000","last_active":"2024-07-26T06:13:36.241645+0000","last_peered":"2024-07-26T06:13:36.241645+0000","last_clean":"2024-07-26T06:13:36.241645+0000","last_became_active":"2024-07-26T06:13:35.064672+0000","last_became_peered":"2024-07-26T06:13:35.064672+0000","last_unstale":"2024-07-26T06:13:36.241645+0000","last_undegraded":"2024-07-26T06:13:36.241645+0000","last_fullsized":"2024-07-26T06:13:36.241645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241131+0000","last_change":"2024-07-26T06:13:35.070018+0000","last_active":"2024-07-26T06:13:36.241131+0000","last_peered":"2024-07-26T06:13:36.241131+0000","last_clean":"2024-07-26T06:13:36.241131+0000","last_became_active":"2024-07-26T06:13:35.069721+0000","last_became_peered":"2024-07-26T06:13:35.069721+0000","last_unstale":"2024-07-26T06:13:36.241131+0000","last_undegraded":"2024-07-26T06:13:36.241131+0000","last_fullsized":"2024-07-26T06:13:36.241131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063011+0000","last_change":"2024-07-26T06:13:35.062741+0000","last_active":"2024-07-26T06:13:36.063011+0000","last_peered":"2024-07-26T06:13:36.063011+0000","last_clean":"2024-07-26T06:13:36.063011+0000","last_became_active":"2024-07-26T06:13:35.061208+0000","last_became_peered":"2024-07-26T06:13:35.061208+0000","last_unstale":"2024-07-26T06:13:36.063011+0000","last_undegraded":"2024-07-26T06:13:36.063011+0000","last_fullsized":"2024-07-26T06:13:36.063011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:50:23.338780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059514+0000","last_change":"2024-07-26T06:13:35.061997+0000","last_active":"2024-07-26T06:13:36.059514+0000","last_peered":"2024-07-26T06:13:36.059514+0000","last_clean":"2024-07-26T06:13:36.059514+0000","last_became_active":"2024-07-26T06:13:35.061691+0000","last_became_peered":"2024-07-26T06:13:35.061691+0000","last_unstale":"2024-07-26T06:13:36.059514+0000","last_undegraded":"2024-07-26T06:13:36.059514+0000","last_fullsized":"2024-07-26T06:13:36.059514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:00.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056756+0000","last_change":"2024-07-26T06:13:35.060288+0000","last_active":"2024-07-26T06:13:36.056756+0000","last_peered":"2024-07-26T06:13:36.056756+0000","last_clean":"2024-07-26T06:13:36.056756+0000","last_became_active":"2024-07-26T06:13:35.059968+0000","last_became_peered":"2024-07-26T06:13:35.059968+0000","last_unstale":"2024-07-26T06:13:36.056756+0000","last_undegraded":"2024-07-26T06:13:36.056756+0000","last_fullsized":"2024-07-26T06:13:36.056756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:23:13.174113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.065640+0000","last_change":"2024-07-26T06:13:35.063974+0000","last_active":"2024-07-26T06:13:35.065640+0000","last_peered":"2024-07-26T06:13:35.065640+0000","last_clean":"2024-07-26T06:13:35.065640+0000","last_became_active":"2024-07-26T06:13:35.062227+0000","last_became_peered":"2024-07-26T06:13:35.062227+0000","last_unstale":"2024-07-26T06:13:35.065640+0000","last_undegraded":"2024-07-26T06:13:35.065640+0000","last_fullsized":"2024-07-26T06:13:35.065640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:38.990666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165749+0000","last_change":"2024-07-26T06:13:35.079657+0000","last_active":"2024-07-26T06:13:36.165749+0000","last_peered":"2024-07-26T06:13:36.165749+0000","last_clean":"2024-07-26T06:13:36.165749+0000","last_became_active":"2024-07-26T06:13:35.078980+0000","last_became_peered":"2024-07-26T06:13:35.078980+0000","last_unstale":"2024-07-26T06:13:36.165749+0000","last_undegraded":"2024-07-26T06:13:36.165749+0000","last_fullsized":"2024-07-26T06:13:36.165749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165976+0000","last_change":"2024-07-26T06:13:35.081733+0000","last_active":"2024-07-26T06:13:36.165976+0000","last_peered":"2024-07-26T06:13:36.165976+0000","last_clean":"2024-07-26T06:13:36.165976+0000","last_became_active":"2024-07-26T06:13:35.081316+0000","last_became_peered":"2024-07-26T06:13:35.081316+0000","last_unstale":"2024-07-26T06:13:36.165976+0000","last_undegraded":"2024-07-26T06:13:36.165976+0000","last_fullsized":"2024-07-26T06:13:36.165976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266057+0000","last_change":"2024-07-26T06:13:35.062888+0000","last_active":"2024-07-26T06:13:37.266057+0000","last_peered":"2024-07-26T06:13:37.266057+0000","last_clean":"2024-07-26T06:13:37.266057+0000","last_became_active":"2024-07-26T06:13:35.062617+0000","last_became_peered":"2024-07-26T06:13:35.062617+0000","last_unstale":"2024-07-26T06:13:37.266057+0000","last_undegraded":"2024-07-26T06:13:37.266057+0000","last_fullsized":"2024-07-26T06:13:37.266057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:28.831320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268031+0000","last_change":"2024-07-26T06:13:35.073326+0000","last_active":"2024-07-26T06:13:37.268031+0000","last_peered":"2024-07-26T06:13:37.268031+0000","last_clean":"2024-07-26T06:13:37.268031+0000","last_became_active":"2024-07-26T06:13:35.073019+0000","last_became_peered":"2024-07-26T06:13:35.073019+0000","last_unstale":"2024-07-26T06:13:37.268031+0000","last_undegraded":"2024-07-26T06:13:37.268031+0000","last_fullsized":"2024-07-26T06:13:37.268031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241158+0000","last_change":"2024-07-26T06:13:35.067632+0000","last_active":"2024-07-26T06:13:36.241158+0000","last_peered":"2024-07-26T06:13:36.241158+0000","last_clean":"2024-07-26T06:13:36.241158+0000","last_became_active":"2024-07-26T06:13:35.066527+0000","last_became_peered":"2024-07-26T06:13:35.066527+0000","last_unstale":"2024-07-26T06:13:36.241158+0000","last_undegraded":"2024-07-26T06:13:36.241158+0000","last_fullsized":"2024-07-26T06:13:36.241158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057001+0000","last_change":"2024-07-26T06:13:35.056477+0000","last_active":"2024-07-26T06:13:36.057001+0000","last_peered":"2024-07-26T06:13:36.057001+0000","last_clean":"2024-07-26T06:13:36.057001+0000","last_became_active":"2024-07-26T06:13:35.056222+0000","last_became_peered":"2024-07-26T06:13:35.056222+0000","last_unstale":"2024-07-26T06:13:36.057001+0000","last_undegraded":"2024-07-26T06:13:36.057001+0000","last_fullsized":"2024-07-26T06:13:36.057001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:20.776784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164819+0000","last_change":"2024-07-26T06:13:35.081680+0000","last_active":"2024-07-26T06:13:36.164819+0000","last_peered":"2024-07-26T06:13:36.164819+0000","last_clean":"2024-07-26T06:13:36.164819+0000","last_became_active":"2024-07-26T06:13:35.081409+0000","last_became_peered":"2024-07-26T06:13:35.081409+0000","last_unstale":"2024-07-26T06:13:36.164819+0000","last_undegraded":"2024-07-26T06:13:36.164819+0000","last_fullsized":"2024-07-26T06:13:36.164819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059747+0000","last_change":"2024-07-26T06:13:35.063598+0000","last_active":"2024-07-26T06:13:36.059747+0000","last_peered":"2024-07-26T06:13:36.059747+0000","last_clean":"2024-07-26T06:13:36.059747+0000","last_became_active":"2024-07-26T06:13:35.063108+0000","last_became_peered":"2024-07-26T06:13:35.063108+0000","last_unstale":"2024-07-26T06:13:36.059747+0000","last_undegraded":"2024-07-26T06:13:36.059747+0000","last_fullsized":"2024-07-26T06:13:36.059747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:49.213504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164852+0000","last_change":"2024-07-26T06:13:35.078539+0000","last_active":"2024-07-26T06:13:36.164852+0000","last_peered":"2024-07-26T06:13:36.164852+0000","last_clean":"2024-07-26T06:13:36.164852+0000","last_became_active":"2024-07-26T06:13:35.074963+0000","last_became_peered":"2024-07-26T06:13:35.074963+0000","last_unstale":"2024-07-26T06:13:36.164852+0000","last_undegraded":"2024-07-26T06:13:36.164852+0000","last_fullsized":"2024-07-26T06:13:36.164852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060557+0000","last_change":"2024-07-26T06:13:35.053573+0000","last_active":"2024-07-26T06:13:36.060557+0000","last_peered":"2024-07-26T06:13:36.060557+0000","last_clean":"2024-07-26T06:13:36.060557+0000","last_became_active":"2024-07-26T06:13:35.053330+0000","last_became_peered":"2024-07-26T06:13:35.053330+0000","last_unstale":"2024-07-26T06:13:36.060557+0000","last_undegraded":"2024-07-26T06:13:36.060557+0000","last_fullsized":"2024-07-26T06:13:36.060557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:01.836541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241629+0000","last_change":"2024-07-26T06:13:35.074100+0000","last_active":"2024-07-26T06:13:36.241629+0000","last_peered":"2024-07-26T06:13:36.241629+0000","last_clean":"2024-07-26T06:13:36.241629+0000","last_became_active":"2024-07-26T06:13:35.073532+0000","last_became_peered":"2024-07-26T06:13:35.073532+0000","last_unstale":"2024-07-26T06:13:36.241629+0000","last_undegraded":"2024-07-26T06:13:36.241629+0000","last_fullsized":"2024-07-26T06:13:36.241629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059866+0000","last_change":"2024-07-26T06:13:35.053991+0000","last_active":"2024-07-26T06:13:35.059866+0000","last_peered":"2024-07-26T06:13:35.059866+0000","last_clean":"2024-07-26T06:13:35.059866+0000","last_became_active":"2024-07-26T06:13:35.053723+0000","last_became_peered":"2024-07-26T06:13:35.053723+0000","last_unstale":"2024-07-26T06:13:35.059866+0000","last_undegraded":"2024-07-26T06:13:35.059866+0000","last_fullsized":"2024-07-26T06:13:35.059866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:38.085872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060277+0000","last_change":"2024-07-26T06:13:35.055687+0000","last_active":"2024-07-26T06:13:36.060277+0000","last_peered":"2024-07-26T06:13:36.060277+0000","last_clean":"2024-07-26T06:13:36.060277+0000","last_became_active":"2024-07-26T06:13:35.054863+0000","last_became_peered":"2024-07-26T06:13:35.054863+0000","last_unstale":"2024-07-26T06:13:36.060277+0000","last_undegraded":"2024-07-26T06:13:36.060277+0000","last_fullsized":"2024-07-26T06:13:36.060277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.827844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059098+0000","last_change":"2024-07-26T06:13:35.047582+0000","last_active":"2024-07-26T06:13:36.059098+0000","last_peered":"2024-07-26T06:13:36.059098+0000","last_clean":"2024-07-26T06:13:36.059098+0000","last_became_active":"2024-07-26T06:13:35.047322+0000","last_became_peered":"2024-07-26T06:13:35.047322+0000","last_unstale":"2024-07-26T06:13:36.059098+0000","last_undegraded":"2024-07-26T06:13:36.059098+0000","last_fullsized":"2024-07-26T06:13:36.059098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165246+0000","last_change":"2024-07-26T06:13:35.079808+0000","last_active":"2024-07-26T06:13:36.165246+0000","last_peered":"2024-07-26T06:13:36.165246+0000","last_clean":"2024-07-26T06:13:36.165246+0000","last_became_active":"2024-07-26T06:13:35.076583+0000","last_became_peered":"2024-07-26T06:13:35.076583+0000","last_unstale":"2024-07-26T06:13:36.165246+0000","last_undegraded":"2024-07-26T06:13:36.165246+0000","last_fullsized":"2024-07-26T06:13:36.165246+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265681+0000","last_change":"2024-07-26T06:13:35.062887+0000","last_active":"2024-07-26T06:13:37.265681+0000","last_peered":"2024-07-26T06:13:37.265681+0000","last_clean":"2024-07-26T06:13:37.265681+0000","last_became_active":"2024-07-26T06:13:35.062553+0000","last_became_peered":"2024-07-26T06:13:35.062553+0000","last_unstale":"2024-07-26T06:13:37.265681+0000","last_undegraded":"2024-07-26T06:13:37.265681+0000","last_fullsized":"2024-07-26T06:13:37.265681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:38.675161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240837+0000","last_change":"2024-07-26T06:13:35.059916+0000","last_active":"2024-07-26T06:13:36.240837+0000","last_peered":"2024-07-26T06:13:36.240837+0000","last_clean":"2024-07-26T06:13:36.240837+0000","last_became_active":"2024-07-26T06:13:35.058276+0000","last_became_peered":"2024-07-26T06:13:35.058276+0000","last_unstale":"2024-07-26T06:13:36.240837+0000","last_undegraded":"2024-07-26T06:13:36.240837+0000","last_fullsized":"2024-07-26T06:13:36.240837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062155+0000","last_change":"2024-07-26T06:13:35.052170+0000","last_active":"2024-07-26T06:13:36.062155+0000","last_peered":"2024-07-26T06:13:36.062155+0000","last_clean":"2024-07-26T06:13:36.062155+0000","last_became_active":"2024-07-26T06:13:35.051630+0000","last_became_peered":"2024-07-26T06:13:35.051630+0000","last_unstale":"2024-07-26T06:13:36.062155+0000","last_undegraded":"2024-07-26T06:13:36.062155+0000","last_fullsized":"2024-07-26T06:13:36.062155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:31.103018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059899+0000","last_change":"2024-07-26T06:13:34.065590+0000","last_active":"2024-07-26T06:13:36.059899+0000","last_peered":"2024-07-26T06:13:36.059899+0000","last_clean":"2024-07-26T06:13:36.059899+0000","last_became_active":"2024-07-26T06:13:34.065275+0000","last_became_peered":"2024-07-26T06:13:34.065275+0000","last_unstale":"2024-07-26T06:13:36.059899+0000","last_undegraded":"2024-07-26T06:13:36.059899+0000","last_fullsized":"2024-07-26T06:13:36.059899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059022+0000","last_change":"2024-07-26T06:13:35.057161+0000","last_active":"2024-07-26T06:13:36.059022+0000","last_peered":"2024-07-26T06:13:36.059022+0000","last_clean":"2024-07-26T06:13:36.059022+0000","last_became_active":"2024-07-26T06:13:35.056843+0000","last_became_peered":"2024-07-26T06:13:35.056843+0000","last_unstale":"2024-07-26T06:13:36.059022+0000","last_undegraded":"2024-07-26T06:13:36.059022+0000","last_fullsized":"2024-07-26T06:13:36.059022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241143+0000","last_change":"2024-07-26T06:13:34.061026+0000","last_active":"2024-07-26T06:13:36.241143+0000","last_peered":"2024-07-26T06:13:36.241143+0000","last_clean":"2024-07-26T06:13:36.241143+0000","last_became_active":"2024-07-26T06:13:34.060729+0000","last_became_peered":"2024-07-26T06:13:34.060729+0000","last_unstale":"2024-07-26T06:13:36.241143+0000","last_undegraded":"2024-07-26T06:13:36.241143+0000","last_fullsized":"2024-07-26T06:13:36.241143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059737+0000","last_change":"2024-07-26T06:13:34.053732+0000","last_active":"2024-07-26T06:13:36.059737+0000","last_peered":"2024-07-26T06:13:36.059737+0000","last_clean":"2024-07-26T06:13:36.059737+0000","last_became_active":"2024-07-26T06:13:34.053385+0000","last_became_peered":"2024-07-26T06:13:34.053385+0000","last_unstale":"2024-07-26T06:13:36.059737+0000","last_undegraded":"2024-07-26T06:13:36.059737+0000","last_fullsized":"2024-07-26T06:13:36.059737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:24:03.458270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059243+0000","last_change":"2024-07-26T06:13:35.042064+0000","last_active":"2024-07-26T06:13:36.059243+0000","last_peered":"2024-07-26T06:13:36.059243+0000","last_clean":"2024-07-26T06:13:36.059243+0000","last_became_active":"2024-07-26T06:13:35.041741+0000","last_became_peered":"2024-07-26T06:13:35.041741+0000","last_unstale":"2024-07-26T06:13:36.059243+0000","last_undegraded":"2024-07-26T06:13:36.059243+0000","last_fullsized":"2024-07-26T06:13:36.059243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:20.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165112+0000","last_change":"2024-07-26T06:13:34.061502+0000","last_active":"2024-07-26T06:13:36.165112+0000","last_peered":"2024-07-26T06:13:36.165112+0000","last_clean":"2024-07-26T06:13:36.165112+0000","last_became_active":"2024-07-26T06:13:34.061155+0000","last_became_peered":"2024-07-26T06:13:34.061155+0000","last_unstale":"2024-07-26T06:13:36.165112+0000","last_undegraded":"2024-07-26T06:13:36.165112+0000","last_fullsized":"2024-07-26T06:13:36.165112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059242+0000","last_change":"2024-07-26T06:13:35.051729+0000","last_active":"2024-07-26T06:13:35.059242+0000","last_peered":"2024-07-26T06:13:35.059242+0000","last_clean":"2024-07-26T06:13:35.059242+0000","last_became_active":"2024-07-26T06:13:35.051429+0000","last_became_peered":"2024-07-26T06:13:35.051429+0000","last_unstale":"2024-07-26T06:13:35.059242+0000","last_undegraded":"2024-07-26T06:13:35.059242+0000","last_fullsized":"2024-07-26T06:13:35.059242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:15:06.230764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241239+0000","last_change":"2024-07-26T06:13:34.066637+0000","last_active":"2024-07-26T06:13:36.241239+0000","last_peered":"2024-07-26T06:13:36.241239+0000","last_clean":"2024-07-26T06:13:36.241239+0000","last_became_active":"2024-07-26T06:13:34.066225+0000","last_became_peered":"2024-07-26T06:13:34.066225+0000","last_unstale":"2024-07-26T06:13:36.241239+0000","last_undegraded":"2024-07-26T06:13:36.241239+0000","last_fullsized":"2024-07-26T06:13:36.241239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165378+0000","last_change":"2024-07-26T06:13:35.080900+0000","last_active":"2024-07-26T06:13:36.165378+0000","last_peered":"2024-07-26T06:13:36.165378+0000","last_clean":"2024-07-26T06:13:36.165378+0000","last_became_active":"2024-07-26T06:13:35.080442+0000","last_became_peered":"2024-07-26T06:13:35.080442+0000","last_unstale":"2024-07-26T06:13:36.165378+0000","last_undegraded":"2024-07-26T06:13:36.165378+0000","last_fullsized":"2024-07-26T06:13:36.165378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165433+0000","last_change":"2024-07-26T06:13:34.060576+0000","last_active":"2024-07-26T06:13:36.165433+0000","last_peered":"2024-07-26T06:13:36.165433+0000","last_clean":"2024-07-26T06:13:36.165433+0000","last_became_active":"2024-07-26T06:13:34.060215+0000","last_became_peered":"2024-07-26T06:13:34.060215+0000","last_unstale":"2024-07-26T06:13:36.165433+0000","last_undegraded":"2024-07-26T06:13:36.165433+0000","last_fullsized":"2024-07-26T06:13:36.165433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:19.760539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057253+0000","last_change":"2024-07-26T06:13:35.072548+0000","last_active":"2024-07-26T06:13:36.057253+0000","last_peered":"2024-07-26T06:13:36.057253+0000","last_clean":"2024-07-26T06:13:36.057253+0000","last_became_active":"2024-07-26T06:13:35.072227+0000","last_became_peered":"2024-07-26T06:13:35.072227+0000","last_unstale":"2024-07-26T06:13:36.057253+0000","last_undegraded":"2024-07-26T06:13:36.057253+0000","last_fullsized":"2024-07-26T06:13:36.057253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.081899+0000","last_change":"2024-07-26T06:13:34.064165+0000","last_active":"2024-07-26T06:13:35.081899+0000","last_peered":"2024-07-26T06:13:35.081899+0000","last_clean":"2024-07-26T06:13:35.081899+0000","last_became_active":"2024-07-26T06:13:34.063828+0000","last_became_peered":"2024-07-26T06:13:34.063828+0000","last_unstale":"2024-07-26T06:13:35.081899+0000","last_undegraded":"2024-07-26T06:13:35.081899+0000","last_fullsized":"2024-07-26T06:13:35.081899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:30.086773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063050+0000","last_change":"2024-07-26T06:13:35.064262+0000","last_active":"2024-07-26T06:13:36.063050+0000","last_peered":"2024-07-26T06:13:36.063050+0000","last_clean":"2024-07-26T06:13:36.063050+0000","last_became_active":"2024-07-26T06:13:35.061569+0000","last_became_peered":"2024-07-26T06:13:35.061569+0000","last_unstale":"2024-07-26T06:13:36.063050+0000","last_undegraded":"2024-07-26T06:13:36.063050+0000","last_fullsized":"2024-07-26T06:13:36.063050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:15:06.230764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241453+0000","last_change":"2024-07-26T06:13:34.053539+0000","last_active":"2024-07-26T06:13:36.241453+0000","last_peered":"2024-07-26T06:13:36.241453+0000","last_clean":"2024-07-26T06:13:36.241453+0000","last_became_active":"2024-07-26T06:13:34.053038+0000","last_became_peered":"2024-07-26T06:13:34.053038+0000","last_unstale":"2024-07-26T06:13:36.241453+0000","last_undegraded":"2024-07-26T06:13:36.241453+0000","last_fullsized":"2024-07-26T06:13:36.241453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267003+0000","last_change":"2024-07-26T06:13:35.075023+0000","last_active":"2024-07-26T06:13:37.267003+0000","last_peered":"2024-07-26T06:13:37.267003+0000","last_clean":"2024-07-26T06:13:37.267003+0000","last_became_active":"2024-07-26T06:13:35.073931+0000","last_became_peered":"2024-07-26T06:13:35.073931+0000","last_unstale":"2024-07-26T06:13:37.267003+0000","last_undegraded":"2024-07-26T06:13:37.267003+0000","last_fullsized":"2024-07-26T06:13:37.267003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059213+0000","last_change":"2024-07-26T06:13:34.048636+0000","last_active":"2024-07-26T06:13:36.059213+0000","last_peered":"2024-07-26T06:13:36.059213+0000","last_clean":"2024-07-26T06:13:36.059213+0000","last_became_active":"2024-07-26T06:13:34.048001+0000","last_became_peered":"2024-07-26T06:13:34.048001+0000","last_unstale":"2024-07-26T06:13:36.059213+0000","last_undegraded":"2024-07-26T06:13:36.059213+0000","last_fullsized":"2024-07-26T06:13:36.059213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:11:14.256545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.058850+0000","last_change":"2024-07-26T06:13:35.057953+0000","last_active":"2024-07-26T06:13:36.058850+0000","last_peered":"2024-07-26T06:13:36.058850+0000","last_clean":"2024-07-26T06:13:36.058850+0000","last_became_active":"2024-07-26T06:13:35.057762+0000","last_became_peered":"2024-07-26T06:13:35.057762+0000","last_unstale":"2024-07-26T06:13:36.058850+0000","last_undegraded":"2024-07-26T06:13:36.058850+0000","last_fullsized":"2024-07-26T06:13:36.058850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165279+0000","last_change":"2024-07-26T06:13:34.058750+0000","last_active":"2024-07-26T06:13:36.165279+0000","last_peered":"2024-07-26T06:13:36.165279+0000","last_clean":"2024-07-26T06:13:36.165279+0000","last_became_active":"2024-07-26T06:13:34.057317+0000","last_became_peered":"2024-07-26T06:13:34.057317+0000","last_unstale":"2024-07-26T06:13:36.165279+0000","last_undegraded":"2024-07-26T06:13:36.165279+0000","last_fullsized":"2024-07-26T06:13:36.165279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267198+0000","last_change":"2024-07-26T06:13:35.068556+0000","last_active":"2024-07-26T06:13:37.267198+0000","last_peered":"2024-07-26T06:13:37.267198+0000","last_clean":"2024-07-26T06:13:37.267198+0000","last_became_active":"2024-07-26T06:13:35.067965+0000","last_became_peered":"2024-07-26T06:13:35.067965+0000","last_unstale":"2024-07-26T06:13:37.267198+0000","last_undegraded":"2024-07-26T06:13:37.267198+0000","last_fullsized":"2024-07-26T06:13:37.267198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241420+0000","last_change":"2024-07-26T06:13:34.062439+0000","last_active":"2024-07-26T06:13:36.241420+0000","last_peered":"2024-07-26T06:13:36.241420+0000","last_clean":"2024-07-26T06:13:36.241420+0000","last_became_active":"2024-07-26T06:13:34.061547+0000","last_became_peered":"2024-07-26T06:13:34.061547+0000","last_unstale":"2024-07-26T06:13:36.241420+0000","last_undegraded":"2024-07-26T06:13:36.241420+0000","last_fullsized":"2024-07-26T06:13:36.241420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240885+0000","last_change":"2024-07-26T06:13:35.059120+0000","last_active":"2024-07-26T06:13:36.240885+0000","last_peered":"2024-07-26T06:13:36.240885+0000","last_clean":"2024-07-26T06:13:36.240885+0000","last_became_active":"2024-07-26T06:13:35.058851+0000","last_became_peered":"2024-07-26T06:13:35.058851+0000","last_unstale":"2024-07-26T06:13:36.240885+0000","last_undegraded":"2024-07-26T06:13:36.240885+0000","last_fullsized":"2024-07-26T06:13:36.240885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265520+0000","last_change":"2024-07-26T06:13:34.052072+0000","last_active":"2024-07-26T06:13:37.265520+0000","last_peered":"2024-07-26T06:13:37.265520+0000","last_clean":"2024-07-26T06:13:37.265520+0000","last_became_active":"2024-07-26T06:13:34.051775+0000","last_became_peered":"2024-07-26T06:13:34.051775+0000","last_unstale":"2024-07-26T06:13:37.265520+0000","last_undegraded":"2024-07-26T06:13:37.265520+0000","last_fullsized":"2024-07-26T06:13:37.265520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266571+0000","last_change":"2024-07-26T06:13:35.069871+0000","last_active":"2024-07-26T06:13:37.266571+0000","last_peered":"2024-07-26T06:13:37.266571+0000","last_clean":"2024-07-26T06:13:37.266571+0000","last_became_active":"2024-07-26T06:13:35.069272+0000","last_became_peered":"2024-07-26T06:13:35.069272+0000","last_unstale":"2024-07-26T06:13:37.266571+0000","last_undegraded":"2024-07-26T06:13:37.266571+0000","last_fullsized":"2024-07-26T06:13:37.266571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062736+0000","last_change":"2024-07-26T06:13:34.050786+0000","last_active":"2024-07-26T06:13:36.062736+0000","last_peered":"2024-07-26T06:13:36.062736+0000","last_clean":"2024-07-26T06:13:36.062736+0000","last_became_active":"2024-07-26T06:13:34.050291+0000","last_became_peered":"2024-07-26T06:13:34.050291+0000","last_unstale":"2024-07-26T06:13:36.062736+0000","last_undegraded":"2024-07-26T06:13:36.062736+0000","last_fullsized":"2024-07-26T06:13:36.062736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:28:47.174016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267602+0000","last_change":"2024-07-26T06:13:35.077176+0000","last_active":"2024-07-26T06:13:37.267602+0000","last_peered":"2024-07-26T06:13:37.267602+0000","last_clean":"2024-07-26T06:13:37.267602+0000","last_became_active":"2024-07-26T06:13:35.074703+0000","last_became_peered":"2024-07-26T06:13:35.074703+0000","last_unstale":"2024-07-26T06:13:37.267602+0000","last_undegraded":"2024-07-26T06:13:37.267602+0000","last_fullsized":"2024-07-26T06:13:37.267602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266222+0000","last_change":"2024-07-26T06:13:34.052622+0000","last_active":"2024-07-26T06:13:37.266222+0000","last_peered":"2024-07-26T06:13:37.266222+0000","last_clean":"2024-07-26T06:13:37.266222+0000","last_became_active":"2024-07-26T06:13:34.052311+0000","last_became_peered":"2024-07-26T06:13:34.052311+0000","last_unstale":"2024-07-26T06:13:37.266222+0000","last_undegraded":"2024-07-26T06:13:37.266222+0000","last_fullsized":"2024-07-26T06:13:37.266222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268227+0000","last_change":"2024-07-26T06:13:35.078765+0000","last_active":"2024-07-26T06:13:37.268227+0000","last_peered":"2024-07-26T06:13:37.268227+0000","last_clean":"2024-07-26T06:13:37.268227+0000","last_became_active":"2024-07-26T06:13:35.077968+0000","last_became_peered":"2024-07-26T06:13:35.077968+0000","last_unstale":"2024-07-26T06:13:37.268227+0000","last_undegraded":"2024-07-26T06:13:37.268227+0000","last_fullsized":"2024-07-26T06:13:37.268227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:22.947178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.067050+0000","last_change":"2024-07-26T06:13:34.047184+0000","last_active":"2024-07-26T06:13:35.067050+0000","last_peered":"2024-07-26T06:13:35.067050+0000","last_clean":"2024-07-26T06:13:35.067050+0000","last_became_active":"2024-07-26T06:13:34.046868+0000","last_became_peered":"2024-07-26T06:13:34.046868+0000","last_unstale":"2024-07-26T06:13:35.067050+0000","last_undegraded":"2024-07-26T06:13:35.067050+0000","last_fullsized":"2024-07-26T06:13:35.067050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:17.313633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165826+0000","last_change":"2024-07-26T06:13:35.065849+0000","last_active":"2024-07-26T06:13:36.165826+0000","last_peered":"2024-07-26T06:13:36.165826+0000","last_clean":"2024-07-26T06:13:36.165826+0000","last_became_active":"2024-07-26T06:13:35.065135+0000","last_became_peered":"2024-07-26T06:13:35.065135+0000","last_unstale":"2024-07-26T06:13:36.165826+0000","last_undegraded":"2024-07-26T06:13:36.165826+0000","last_fullsized":"2024-07-26T06:13:36.165826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265761+0000","last_change":"2024-07-26T06:13:34.062785+0000","last_active":"2024-07-26T06:13:37.265761+0000","last_peered":"2024-07-26T06:13:37.265761+0000","last_clean":"2024-07-26T06:13:37.265761+0000","last_became_active":"2024-07-26T06:13:34.062466+0000","last_became_peered":"2024-07-26T06:13:34.062466+0000","last_unstale":"2024-07-26T06:13:37.265761+0000","last_undegraded":"2024-07-26T06:13:37.265761+0000","last_fullsized":"2024-07-26T06:13:37.265761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:13:51.856008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267453+0000","last_change":"2024-07-26T06:13:35.082619+0000","last_active":"2024-07-26T06:13:37.267453+0000","last_peered":"2024-07-26T06:13:37.267453+0000","last_clean":"2024-07-26T06:13:37.267453+0000","last_became_active":"2024-07-26T06:13:35.082044+0000","last_became_peered":"2024-07-26T06:13:35.082044+0000","last_unstale":"2024-07-26T06:13:37.267453+0000","last_undegraded":"2024-07-26T06:13:37.267453+0000","last_fullsized":"2024-07-26T06:13:37.267453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266471+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:13:37.266471+0000","last_peered":"2024-07-26T06:13:37.266471+0000","last_clean":"2024-07-26T06:13:37.266471+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:13:37.266471+0000","last_undegraded":"2024-07-26T06:13:37.266471+0000","last_fullsized":"2024-07-26T06:13:37.266471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164685+0000","last_change":"2024-07-26T06:13:34.061955+0000","last_active":"2024-07-26T06:13:36.164685+0000","last_peered":"2024-07-26T06:13:36.164685+0000","last_clean":"2024-07-26T06:13:36.164685+0000","last_became_active":"2024-07-26T06:13:34.060172+0000","last_became_peered":"2024-07-26T06:13:34.060172+0000","last_unstale":"2024-07-26T06:13:36.164685+0000","last_undegraded":"2024-07-26T06:13:36.164685+0000","last_fullsized":"2024-07-26T06:13:36.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:45.461779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059576+0000","last_change":"2024-07-26T06:13:35.077259+0000","last_active":"2024-07-26T06:13:36.059576+0000","last_peered":"2024-07-26T06:13:36.059576+0000","last_clean":"2024-07-26T06:13:36.059576+0000","last_became_active":"2024-07-26T06:13:35.076879+0000","last_became_peered":"2024-07-26T06:13:35.076879+0000","last_unstale":"2024-07-26T06:13:36.059576+0000","last_undegraded":"2024-07-26T06:13:36.059576+0000","last_fullsized":"2024-07-26T06:13:36.059576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:27.327738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266356+0000","last_change":"2024-07-26T06:13:34.064601+0000","last_active":"2024-07-26T06:13:37.266356+0000","last_peered":"2024-07-26T06:13:37.266356+0000","last_clean":"2024-07-26T06:13:37.266356+0000","last_became_active":"2024-07-26T06:13:34.064270+0000","last_became_peered":"2024-07-26T06:13:34.064270+0000","last_unstale":"2024-07-26T06:13:37.266356+0000","last_undegraded":"2024-07-26T06:13:37.266356+0000","last_fullsized":"2024-07-26T06:13:37.266356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056647+0000","last_change":"2024-07-26T06:13:35.062026+0000","last_active":"2024-07-26T06:13:36.056647+0000","last_peered":"2024-07-26T06:13:36.056647+0000","last_clean":"2024-07-26T06:13:36.056647+0000","last_became_active":"2024-07-26T06:13:35.061732+0000","last_became_peered":"2024-07-26T06:13:35.061732+0000","last_unstale":"2024-07-26T06:13:36.056647+0000","last_undegraded":"2024-07-26T06:13:36.056647+0000","last_fullsized":"2024-07-26T06:13:36.056647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:32.385097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265755+0000","last_change":"2024-07-26T06:13:34.057752+0000","last_active":"2024-07-26T06:13:37.265755+0000","last_peered":"2024-07-26T06:13:37.265755+0000","last_clean":"2024-07-26T06:13:37.265755+0000","last_became_active":"2024-07-26T06:13:34.057421+0000","last_became_peered":"2024-07-26T06:13:34.057421+0000","last_unstale":"2024-07-26T06:13:37.265755+0000","last_undegraded":"2024-07-26T06:13:37.265755+0000","last_fullsized":"2024-07-26T06:13:37.265755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059275+0000","last_change":"2024-07-26T06:13:35.051454+0000","last_active":"2024-07-26T06:13:35.059275+0000","last_peered":"2024-07-26T06:13:35.059275+0000","last_clean":"2024-07-26T06:13:35.059275+0000","last_became_active":"2024-07-26T06:13:35.051175+0000","last_became_peered":"2024-07-26T06:13:35.051175+0000","last_unstale":"2024-07-26T06:13:35.059275+0000","last_undegraded":"2024-07-26T06:13:35.059275+0000","last_fullsized":"2024-07-26T06:13:35.059275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:50:23.338780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165555+0000","last_change":"2024-07-26T06:13:34.055084+0000","last_active":"2024-07-26T06:13:36.165555+0000","last_peered":"2024-07-26T06:13:36.165555+0000","last_clean":"2024-07-26T06:13:36.165555+0000","last_became_active":"2024-07-26T06:13:34.054390+0000","last_became_peered":"2024-07-26T06:13:34.054390+0000","last_unstale":"2024-07-26T06:13:36.165555+0000","last_undegraded":"2024-07-26T06:13:36.165555+0000","last_fullsized":"2024-07-26T06:13:36.165555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062792+0000","last_change":"2024-07-26T06:13:35.052783+0000","last_active":"2024-07-26T06:13:36.062792+0000","last_peered":"2024-07-26T06:13:36.062792+0000","last_clean":"2024-07-26T06:13:36.062792+0000","last_became_active":"2024-07-26T06:13:35.048901+0000","last_became_peered":"2024-07-26T06:13:35.048901+0000","last_unstale":"2024-07-26T06:13:36.062792+0000","last_undegraded":"2024-07-26T06:13:36.062792+0000","last_fullsized":"2024-07-26T06:13:36.062792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:18.329878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.049721+0000","last_change":"2024-07-26T06:13:34.049038+0000","last_active":"2024-07-26T06:13:35.049721+0000","last_peered":"2024-07-26T06:13:35.049721+0000","last_clean":"2024-07-26T06:13:35.049721+0000","last_became_active":"2024-07-26T06:13:34.048751+0000","last_became_peered":"2024-07-26T06:13:34.048751+0000","last_unstale":"2024-07-26T06:13:35.049721+0000","last_undegraded":"2024-07-26T06:13:35.049721+0000","last_fullsized":"2024-07-26T06:13:35.049721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165618+0000","last_change":"2024-07-26T06:13:35.080209+0000","last_active":"2024-07-26T06:13:36.165618+0000","last_peered":"2024-07-26T06:13:36.165618+0000","last_clean":"2024-07-26T06:13:36.165618+0000","last_became_active":"2024-07-26T06:13:35.079870+0000","last_became_peered":"2024-07-26T06:13:35.079870+0000","last_unstale":"2024-07-26T06:13:36.165618+0000","last_undegraded":"2024-07-26T06:13:36.165618+0000","last_fullsized":"2024-07-26T06:13:36.165618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268083+0000","last_change":"2024-07-26T06:13:34.066620+0000","last_active":"2024-07-26T06:13:37.268083+0000","last_peered":"2024-07-26T06:13:37.268083+0000","last_clean":"2024-07-26T06:13:37.268083+0000","last_became_active":"2024-07-26T06:13:34.066296+0000","last_became_peered":"2024-07-26T06:13:34.066296+0000","last_unstale":"2024-07-26T06:13:37.268083+0000","last_undegraded":"2024-07-26T06:13:37.268083+0000","last_fullsized":"2024-07-26T06:13:37.268083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:09:26.010192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059439+0000","last_change":"2024-07-26T06:13:35.052176+0000","last_active":"2024-07-26T06:13:35.059439+0000","last_peered":"2024-07-26T06:13:35.059439+0000","last_clean":"2024-07-26T06:13:35.059439+0000","last_became_active":"2024-07-26T06:13:35.051905+0000","last_became_peered":"2024-07-26T06:13:35.051905+0000","last_unstale":"2024-07-26T06:13:35.059439+0000","last_undegraded":"2024-07-26T06:13:35.059439+0000","last_fullsized":"2024-07-26T06:13:35.059439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:41.825450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.049853+0000","last_change":"2024-07-26T06:13:34.059549+0000","last_active":"2024-07-26T06:13:35.049853+0000","last_peered":"2024-07-26T06:13:35.049853+0000","last_clean":"2024-07-26T06:13:35.049853+0000","last_became_active":"2024-07-26T06:13:34.058721+0000","last_became_peered":"2024-07-26T06:13:34.058721+0000","last_unstale":"2024-07-26T06:13:35.049853+0000","last_undegraded":"2024-07-26T06:13:35.049853+0000","last_fullsized":"2024-07-26T06:13:35.049853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060701+0000","last_change":"2024-07-26T06:13:35.066322+0000","last_active":"2024-07-26T06:13:36.060701+0000","last_peered":"2024-07-26T06:13:36.060701+0000","last_clean":"2024-07-26T06:13:36.060701+0000","last_became_active":"2024-07-26T06:13:35.066006+0000","last_became_peered":"2024-07-26T06:13:35.066006+0000","last_unstale":"2024-07-26T06:13:36.060701+0000","last_undegraded":"2024-07-26T06:13:36.060701+0000","last_fullsized":"2024-07-26T06:13:36.060701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:31.900907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241473+0000","last_change":"2024-07-26T06:13:34.069982+0000","last_active":"2024-07-26T06:13:36.241473+0000","last_peered":"2024-07-26T06:13:36.241473+0000","last_clean":"2024-07-26T06:13:36.241473+0000","last_became_active":"2024-07-26T06:13:34.069000+0000","last_became_peered":"2024-07-26T06:13:34.069000+0000","last_unstale":"2024-07-26T06:13:36.241473+0000","last_undegraded":"2024-07-26T06:13:36.241473+0000","last_fullsized":"2024-07-26T06:13:36.241473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059318+0000","last_change":"2024-07-26T06:13:35.059073+0000","last_active":"2024-07-26T06:13:36.059318+0000","last_peered":"2024-07-26T06:13:36.059318+0000","last_clean":"2024-07-26T06:13:36.059318+0000","last_became_active":"2024-07-26T06:13:35.058724+0000","last_became_peered":"2024-07-26T06:13:35.058724+0000","last_unstale":"2024-07-26T06:13:36.059318+0000","last_undegraded":"2024-07-26T06:13:36.059318+0000","last_fullsized":"2024-07-26T06:13:36.059318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266310+0000","last_change":"2024-07-26T06:13:34.065271+0000","last_active":"2024-07-26T06:13:37.266310+0000","last_peered":"2024-07-26T06:13:37.266310+0000","last_clean":"2024-07-26T06:13:37.266310+0000","last_became_active":"2024-07-26T06:13:34.064922+0000","last_became_peered":"2024-07-26T06:13:34.064922+0000","last_unstale":"2024-07-26T06:13:37.266310+0000","last_undegraded":"2024-07-26T06:13:37.266310+0000","last_fullsized":"2024-07-26T06:13:37.266310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062133+0000","last_change":"2024-07-26T06:13:35.056227+0000","last_active":"2024-07-26T06:13:36.062133+0000","last_peered":"2024-07-26T06:13:36.062133+0000","last_clean":"2024-07-26T06:13:36.062133+0000","last_became_active":"2024-07-26T06:13:35.055855+0000","last_became_peered":"2024-07-26T06:13:35.055855+0000","last_unstale":"2024-07-26T06:13:36.062133+0000","last_undegraded":"2024-07-26T06:13:36.062133+0000","last_fullsized":"2024-07-26T06:13:36.062133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:03.118475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.050762+0000","last_change":"2024-07-26T06:13:34.066397+0000","last_active":"2024-07-26T06:13:35.050762+0000","last_peered":"2024-07-26T06:13:35.050762+0000","last_clean":"2024-07-26T06:13:35.050762+0000","last_became_active":"2024-07-26T06:13:34.066084+0000","last_became_peered":"2024-07-26T06:13:34.066084+0000","last_unstale":"2024-07-26T06:13:35.050762+0000","last_undegraded":"2024-07-26T06:13:35.050762+0000","last_fullsized":"2024-07-26T06:13:35.050762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:02.102230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059415+0000","last_change":"2024-07-26T06:13:35.053587+0000","last_active":"2024-07-26T06:13:36.059415+0000","last_peered":"2024-07-26T06:13:36.059415+0000","last_clean":"2024-07-26T06:13:36.059415+0000","last_became_active":"2024-07-26T06:13:35.053237+0000","last_became_peered":"2024-07-26T06:13:35.053237+0000","last_unstale":"2024-07-26T06:13:36.059415+0000","last_undegraded":"2024-07-26T06:13:36.059415+0000","last_fullsized":"2024-07-26T06:13:36.059415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:07.936415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.050114+0000","last_change":"2024-07-26T06:13:34.044567+0000","last_active":"2024-07-26T06:13:35.050114+0000","last_peered":"2024-07-26T06:13:35.050114+0000","last_clean":"2024-07-26T06:13:35.050114+0000","last_became_active":"2024-07-26T06:13:34.043889+0000","last_became_peered":"2024-07-26T06:13:34.043889+0000","last_unstale":"2024-07-26T06:13:35.050114+0000","last_undegraded":"2024-07-26T06:13:35.050114+0000","last_fullsized":"2024-07-26T06:13:35.050114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:40.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060003+0000","last_change":"2024-07-26T06:13:35.043595+0000","last_active":"2024-07-26T06:13:36.060003+0000","last_peered":"2024-07-26T06:13:36.060003+0000","last_clean":"2024-07-26T06:13:36.060003+0000","last_became_active":"2024-07-26T06:13:35.043258+0000","last_became_peered":"2024-07-26T06:13:35.043258+0000","last_unstale":"2024-07-26T06:13:36.060003+0000","last_undegraded":"2024-07-26T06:13:36.060003+0000","last_fullsized":"2024-07-26T06:13:36.060003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:48.040021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267509+0000","last_change":"2024-07-26T06:13:34.064636+0000","last_active":"2024-07-26T06:13:37.267509+0000","last_peered":"2024-07-26T06:13:37.267509+0000","last_clean":"2024-07-26T06:13:37.267509+0000","last_became_active":"2024-07-26T06:13:34.064308+0000","last_became_peered":"2024-07-26T06:13:34.064308+0000","last_unstale":"2024-07-26T06:13:37.267509+0000","last_undegraded":"2024-07-26T06:13:37.267509+0000","last_fullsized":"2024-07-26T06:13:37.267509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240857+0000","last_change":"2024-07-26T06:13:35.066830+0000","last_active":"2024-07-26T06:13:36.240857+0000","last_peered":"2024-07-26T06:13:36.240857+0000","last_clean":"2024-07-26T06:13:36.240857+0000","last_became_active":"2024-07-26T06:13:35.066494+0000","last_became_peered":"2024-07-26T06:13:35.066494+0000","last_unstale":"2024-07-26T06:13:36.240857+0000","last_undegraded":"2024-07-26T06:13:36.240857+0000","last_fullsized":"2024-07-26T06:13:36.240857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165479+0000","last_change":"2024-07-26T06:13:35.079246+0000","last_active":"2024-07-26T06:13:36.165479+0000","last_peered":"2024-07-26T06:13:36.165479+0000","last_clean":"2024-07-26T06:13:36.165479+0000","last_became_active":"2024-07-26T06:13:35.075982+0000","last_became_peered":"2024-07-26T06:13:35.075982+0000","last_unstale":"2024-07-26T06:13:36.165479+0000","last_undegraded":"2024-07-26T06:13:36.165479+0000","last_fullsized":"2024-07-26T06:13:36.165479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267123+0000","last_change":"2024-07-26T06:13:34.066151+0000","last_active":"2024-07-26T06:13:37.267123+0000","last_peered":"2024-07-26T06:13:37.267123+0000","last_clean":"2024-07-26T06:13:37.267123+0000","last_became_active":"2024-07-26T06:13:34.065737+0000","last_became_peered":"2024-07-26T06:13:34.065737+0000","last_unstale":"2024-07-26T06:13:37.267123+0000","last_undegraded":"2024-07-26T06:13:37.267123+0000","last_fullsized":"2024-07-26T06:13:37.267123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165512+0000","last_change":"2024-07-26T06:13:35.075547+0000","last_active":"2024-07-26T06:13:36.165512+0000","last_peered":"2024-07-26T06:13:36.165512+0000","last_clean":"2024-07-26T06:13:36.165512+0000","last_became_active":"2024-07-26T06:13:35.075262+0000","last_became_peered":"2024-07-26T06:13:35.075262+0000","last_unstale":"2024-07-26T06:13:36.165512+0000","last_undegraded":"2024-07-26T06:13:36.165512+0000","last_fullsized":"2024-07-26T06:13:36.165512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267491+0000","last_change":"2024-07-26T06:13:34.066478+0000","last_active":"2024-07-26T06:13:37.267491+0000","last_peered":"2024-07-26T06:13:37.267491+0000","last_clean":"2024-07-26T06:13:37.267491+0000","last_became_active":"2024-07-26T06:13:34.066015+0000","last_became_peered":"2024-07-26T06:13:34.066015+0000","last_unstale":"2024-07-26T06:13:37.267491+0000","last_undegraded":"2024-07-26T06:13:37.267491+0000","last_fullsized":"2024-07-26T06:13:37.267491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:27.815076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059941+0000","last_change":"2024-07-26T06:13:35.077607+0000","last_active":"2024-07-26T06:13:36.059941+0000","last_peered":"2024-07-26T06:13:36.059941+0000","last_clean":"2024-07-26T06:13:36.059941+0000","last_became_active":"2024-07-26T06:13:35.077423+0000","last_became_peered":"2024-07-26T06:13:35.077423+0000","last_unstale":"2024-07-26T06:13:36.059941+0000","last_undegraded":"2024-07-26T06:13:36.059941+0000","last_fullsized":"2024-07-26T06:13:36.059941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060485+0000","last_change":"2024-07-26T06:13:34.052811+0000","last_active":"2024-07-26T06:13:36.060485+0000","last_peered":"2024-07-26T06:13:36.060485+0000","last_clean":"2024-07-26T06:13:36.060485+0000","last_became_active":"2024-07-26T06:13:34.051989+0000","last_became_peered":"2024-07-26T06:13:34.051989+0000","last_unstale":"2024-07-26T06:13:36.060485+0000","last_undegraded":"2024-07-26T06:13:36.060485+0000","last_fullsized":"2024-07-26T06:13:36.060485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060313+0000","last_change":"2024-07-26T06:13:35.061621+0000","last_active":"2024-07-26T06:13:36.060313+0000","last_peered":"2024-07-26T06:13:36.060313+0000","last_clean":"2024-07-26T06:13:36.060313+0000","last_became_active":"2024-07-26T06:13:35.061349+0000","last_became_peered":"2024-07-26T06:13:35.061349+0000","last_unstale":"2024-07-26T06:13:36.060313+0000","last_undegraded":"2024-07-26T06:13:36.060313+0000","last_fullsized":"2024-07-26T06:13:36.060313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:06:30.885224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059607+0000","last_change":"2024-07-26T06:13:34.069215+0000","last_active":"2024-07-26T06:13:36.059607+0000","last_peered":"2024-07-26T06:13:36.059607+0000","last_clean":"2024-07-26T06:13:36.059607+0000","last_became_active":"2024-07-26T06:13:34.068883+0000","last_became_peered":"2024-07-26T06:13:34.068883+0000","last_unstale":"2024-07-26T06:13:36.059607+0000","last_undegraded":"2024-07-26T06:13:36.059607+0000","last_fullsized":"2024-07-26T06:13:36.059607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:02:49.446347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":657191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":66367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:13:39.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:13:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97e00640 1 -- 172.21.15.163:0/1509706707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 msgr2=0x7fce7c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97e00640 1 --2- 172.21.15.163:0/1509706707 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce7c004c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fce980712b0 tx=0x7fce7c005830 comp rx=0 tx=0).stop 2024-07-26T06:13:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97400640 1 -- 172.21.15.163:0/1509706707 reap_dead start 2024-07-26T06:13:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97e00640 1 -- 172.21.15.163:0/1509706707 shutdown_connections 2024-07-26T06:13:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97e00640 1 -- 172.21.15.163:0/1509706707 >> 172.21.15.163:0/1509706707 conn(0x7fce980b0e30 msgr2=0x7fce980b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97e00640 1 -- 172.21.15.163:0/1509706707 shutdown_connections 2024-07-26T06:13:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.709+0000 7fce97e00640 1 -- 172.21.15.163:0/1509706707 wait complete. 2024-07-26T06:13:39.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.711+0000 7fce97e00640 1 Processor -- start 2024-07-26T06:13:39.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.711+0000 7fce97e00640 1 -- start start 2024-07-26T06:13:39.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce97e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce98072270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce97e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce98146140 0x7fce98073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce97e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce98074500 0x7fce9814f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce97e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fce9815d560 con 0x7fce7c004800 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce97e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce9806f500 con 0x7fce98146140 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce97e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fce9815d6e0 con 0x7fce98074500 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce96a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce98072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce96a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce98072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44860/0 (socket says 172.21.15.163:44860) 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.712+0000 7fce96a00640 1 -- 172.21.15.163:0/3212467313 learned_addr learned my addr 172.21.15.163:0/3212467313 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce97400640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce98074500 0x7fce9814f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96000640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce98146140 0x7fce98073fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96a00640 1 -- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce98074500 msgr2=0x7fce9814f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96a00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce98074500 0x7fce9814f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96a00640 1 -- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce98146140 msgr2=0x7fce98073fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96a00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce98146140 0x7fce98073fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96a00640 1 -- 172.21.15.163:0/3212467313 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce98072c40 con 0x7fce7c004800 2024-07-26T06:13:39.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce96a00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce98072270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fce8800fb10 tx=0x7fce8800d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:39.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88004b20 con 0x7fce7c004800 2024-07-26T06:13:39.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce88004cc0 con 0x7fce7c004800 2024-07-26T06:13:39.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce88005280 con 0x7fce7c004800 2024-07-26T06:13:39.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce98072f20 con 0x7fce7c004800 2024-07-26T06:13:39.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.713+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce980734e0 con 0x7fce7c004800 2024-07-26T06:13:39.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.715+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce98159750 con 0x7fce7c004800 2024-07-26T06:13:39.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.715+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fce8800c030 con 0x7fce7c004800 2024-07-26T06:13:39.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.716+0000 7fce8f400640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fce780762d0 0x7fce78078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:39.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.717+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fce88098240 con 0x7fce7c004800 2024-07-26T06:13:39.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.717+0000 7fce96000640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fce780762d0 0x7fce78078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:39.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.718+0000 7fce96000640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fce780762d0 0x7fce78078790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce84036000 tx=0x7fce840046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:39.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.720+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fce88061b00 con 0x7fce7c004800 2024-07-26T06:13:39.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.923+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce9806d400 con 0x7fce780762d0 2024-07-26T06:13:39.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.936+0000 7fce8f400640 1 -- 172.21.15.163:0/3212467313 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251929 (secure 0 0 0) 0x7fce9806d400 con 0x7fce780762d0 2024-07-26T06:13:39.941 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:39.942 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fce780762d0 msgr2=0x7fce78078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fce780762d0 0x7fce78078790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce84036000 tx=0x7fce840046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 msgr2=0x7fce98072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce98072270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fce8800fb10 tx=0x7fce8800d2e0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 shutdown_connections 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fce780762d0 0x7fce78078790 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce98074500 0x7fce9814f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce98146140 0x7fce98073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 --2- 172.21.15.163:0/3212467313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce7c004800 0x7fce98072270 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 >> 172.21.15.163:0/3212467313 conn(0x7fce980b0e30 msgr2=0x7fce98155010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 shutdown_connections 2024-07-26T06:13:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:39.943+0000 7fce97e00640 1 -- 172.21.15.163:0/3212467313 wait complete. 2024-07-26T06:13:39.964 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-07-26T06:13:39.645590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218984,"kb_used_data":2808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755736,"statfs":{"total":773094113280,"available":772869873664,"internally_reserved":0,"allocated":2875392,"data_stored":1740684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002207"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267640+0000","last_change":"2024-07-26T06:13:35.072083+0000","last_active":"2024-07-26T06:13:37.267640+0000","last_peered":"2024-07-26T06:13:37.267640+0000","last_clean":"2024-07-26T06:13:37.267640+0000","last_became_active":"2024-07-26T06:13:35.071746+0000","last_became_peered":"2024-07-26T06:13:35.071746+0000","last_unstale":"2024-07-26T06:13:37.267640+0000","last_undegraded":"2024-07-26T06:13:37.267640+0000","last_fullsized":"2024-07-26T06:13:37.267640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.063250+0000","last_change":"2024-07-26T06:13:35.058738+0000","last_active":"2024-07-26T06:13:35.063250+0000","last_peered":"2024-07-26T06:13:35.063250+0000","last_clean":"2024-07-26T06:13:35.063250+0000","last_became_active":"2024-07-26T06:13:35.058470+0000","last_became_peered":"2024-07-26T06:13:35.058470+0000","last_unstale":"2024-07-26T06:13:35.063250+0000","last_undegraded":"2024-07-26T06:13:35.063250+0000","last_fullsized":"2024-07-26T06:13:35.063250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.331691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056891+0000","last_change":"2024-07-26T06:13:35.042867+0000","last_active":"2024-07-26T06:13:36.056891+0000","last_peered":"2024-07-26T06:13:36.056891+0000","last_clean":"2024-07-26T06:13:36.056891+0000","last_became_active":"2024-07-26T06:13:35.042660+0000","last_became_peered":"2024-07-26T06:13:35.042660+0000","last_unstale":"2024-07-26T06:13:36.056891+0000","last_undegraded":"2024-07-26T06:13:36.056891+0000","last_fullsized":"2024-07-26T06:13:36.056891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:17:02.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165050+0000","last_change":"2024-07-26T06:13:35.076112+0000","last_active":"2024-07-26T06:13:36.165050+0000","last_peered":"2024-07-26T06:13:36.165050+0000","last_clean":"2024-07-26T06:13:36.165050+0000","last_became_active":"2024-07-26T06:13:35.075903+0000","last_became_peered":"2024-07-26T06:13:35.075903+0000","last_unstale":"2024-07-26T06:13:36.165050+0000","last_undegraded":"2024-07-26T06:13:36.165050+0000","last_fullsized":"2024-07-26T06:13:36.165050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267271+0000","last_change":"2024-07-26T06:13:35.068273+0000","last_active":"2024-07-26T06:13:37.267271+0000","last_peered":"2024-07-26T06:13:37.267271+0000","last_clean":"2024-07-26T06:13:37.267271+0000","last_became_active":"2024-07-26T06:13:35.067645+0000","last_became_peered":"2024-07-26T06:13:35.067645+0000","last_unstale":"2024-07-26T06:13:37.267271+0000","last_undegraded":"2024-07-26T06:13:37.267271+0000","last_fullsized":"2024-07-26T06:13:37.267271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164727+0000","last_change":"2024-07-26T06:13:35.080356+0000","last_active":"2024-07-26T06:13:36.164727+0000","last_peered":"2024-07-26T06:13:36.164727+0000","last_clean":"2024-07-26T06:13:36.164727+0000","last_became_active":"2024-07-26T06:13:35.079906+0000","last_became_peered":"2024-07-26T06:13:35.079906+0000","last_unstale":"2024-07-26T06:13:36.164727+0000","last_undegraded":"2024-07-26T06:13:36.164727+0000","last_fullsized":"2024-07-26T06:13:36.164727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266764+0000","last_change":"2024-07-26T06:13:35.070227+0000","last_active":"2024-07-26T06:13:37.266764+0000","last_peered":"2024-07-26T06:13:37.266764+0000","last_clean":"2024-07-26T06:13:37.266764+0000","last_became_active":"2024-07-26T06:13:35.069602+0000","last_became_peered":"2024-07-26T06:13:35.069602+0000","last_unstale":"2024-07-26T06:13:37.266764+0000","last_undegraded":"2024-07-26T06:13:37.266764+0000","last_fullsized":"2024-07-26T06:13:37.266764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059249+0000","last_change":"2024-07-26T06:13:35.077098+0000","last_active":"2024-07-26T06:13:36.059249+0000","last_peered":"2024-07-26T06:13:36.059249+0000","last_clean":"2024-07-26T06:13:36.059249+0000","last_became_active":"2024-07-26T06:13:35.076814+0000","last_became_peered":"2024-07-26T06:13:35.076814+0000","last_unstale":"2024-07-26T06:13:36.059249+0000","last_undegraded":"2024-07-26T06:13:36.059249+0000","last_fullsized":"2024-07-26T06:13:36.059249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:19.325249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267856+0000","last_change":"2024-07-26T06:13:35.081415+0000","last_active":"2024-07-26T06:13:37.267856+0000","last_peered":"2024-07-26T06:13:37.267856+0000","last_clean":"2024-07-26T06:13:37.267856+0000","last_became_active":"2024-07-26T06:13:35.080563+0000","last_became_peered":"2024-07-26T06:13:35.080563+0000","last_unstale":"2024-07-26T06:13:37.267856+0000","last_undegraded":"2024-07-26T06:13:37.267856+0000","last_fullsized":"2024-07-26T06:13:37.267856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241645+0000","last_change":"2024-07-26T06:13:35.067607+0000","last_active":"2024-07-26T06:13:36.241645+0000","last_peered":"2024-07-26T06:13:36.241645+0000","last_clean":"2024-07-26T06:13:36.241645+0000","last_became_active":"2024-07-26T06:13:35.064672+0000","last_became_peered":"2024-07-26T06:13:35.064672+0000","last_unstale":"2024-07-26T06:13:36.241645+0000","last_undegraded":"2024-07-26T06:13:36.241645+0000","last_fullsized":"2024-07-26T06:13:36.241645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241131+0000","last_change":"2024-07-26T06:13:35.070018+0000","last_active":"2024-07-26T06:13:36.241131+0000","last_peered":"2024-07-26T06:13:36.241131+0000","last_clean":"2024-07-26T06:13:36.241131+0000","last_became_active":"2024-07-26T06:13:35.069721+0000","last_became_peered":"2024-07-26T06:13:35.069721+0000","last_unstale":"2024-07-26T06:13:36.241131+0000","last_undegraded":"2024-07-26T06:13:36.241131+0000","last_fullsized":"2024-07-26T06:13:36.241131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063011+0000","last_change":"2024-07-26T06:13:35.062741+0000","last_active":"2024-07-26T06:13:36.063011+0000","last_peered":"2024-07-26T06:13:36.063011+0000","last_clean":"2024-07-26T06:13:36.063011+0000","last_became_active":"2024-07-26T06:13:35.061208+0000","last_became_peered":"2024-07-26T06:13:35.061208+0000","last_unstale":"2024-07-26T06:13:36.063011+0000","last_undegraded":"2024-07-26T06:13:36.063011+0000","last_fullsized":"2024-07-26T06:13:36.063011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:50:23.338780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059514+0000","last_change":"2024-07-26T06:13:35.061997+0000","last_active":"2024-07-26T06:13:36.059514+0000","last_peered":"2024-07-26T06:13:36.059514+0000","last_clean":"2024-07-26T06:13:36.059514+0000","last_became_active":"2024-07-26T06:13:35.061691+0000","last_became_peered":"2024-07-26T06:13:35.061691+0000","last_unstale":"2024-07-26T06:13:36.059514+0000","last_undegraded":"2024-07-26T06:13:36.059514+0000","last_fullsized":"2024-07-26T06:13:36.059514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:00.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056756+0000","last_change":"2024-07-26T06:13:35.060288+0000","last_active":"2024-07-26T06:13:36.056756+0000","last_peered":"2024-07-26T06:13:36.056756+0000","last_clean":"2024-07-26T06:13:36.056756+0000","last_became_active":"2024-07-26T06:13:35.059968+0000","last_became_peered":"2024-07-26T06:13:35.059968+0000","last_unstale":"2024-07-26T06:13:36.056756+0000","last_undegraded":"2024-07-26T06:13:36.056756+0000","last_fullsized":"2024-07-26T06:13:36.056756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:23:13.174113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.065640+0000","last_change":"2024-07-26T06:13:35.063974+0000","last_active":"2024-07-26T06:13:35.065640+0000","last_peered":"2024-07-26T06:13:35.065640+0000","last_clean":"2024-07-26T06:13:35.065640+0000","last_became_active":"2024-07-26T06:13:35.062227+0000","last_became_peered":"2024-07-26T06:13:35.062227+0000","last_unstale":"2024-07-26T06:13:35.065640+0000","last_undegraded":"2024-07-26T06:13:35.065640+0000","last_fullsized":"2024-07-26T06:13:35.065640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:38.990666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165749+0000","last_change":"2024-07-26T06:13:35.079657+0000","last_active":"2024-07-26T06:13:36.165749+0000","last_peered":"2024-07-26T06:13:36.165749+0000","last_clean":"2024-07-26T06:13:36.165749+0000","last_became_active":"2024-07-26T06:13:35.078980+0000","last_became_peered":"2024-07-26T06:13:35.078980+0000","last_unstale":"2024-07-26T06:13:36.165749+0000","last_undegraded":"2024-07-26T06:13:36.165749+0000","last_fullsized":"2024-07-26T06:13:36.165749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165976+0000","last_change":"2024-07-26T06:13:35.081733+0000","last_active":"2024-07-26T06:13:36.165976+0000","last_peered":"2024-07-26T06:13:36.165976+0000","last_clean":"2024-07-26T06:13:36.165976+0000","last_became_active":"2024-07-26T06:13:35.081316+0000","last_became_peered":"2024-07-26T06:13:35.081316+0000","last_unstale":"2024-07-26T06:13:36.165976+0000","last_undegraded":"2024-07-26T06:13:36.165976+0000","last_fullsized":"2024-07-26T06:13:36.165976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266057+0000","last_change":"2024-07-26T06:13:35.062888+0000","last_active":"2024-07-26T06:13:37.266057+0000","last_peered":"2024-07-26T06:13:37.266057+0000","last_clean":"2024-07-26T06:13:37.266057+0000","last_became_active":"2024-07-26T06:13:35.062617+0000","last_became_peered":"2024-07-26T06:13:35.062617+0000","last_unstale":"2024-07-26T06:13:37.266057+0000","last_undegraded":"2024-07-26T06:13:37.266057+0000","last_fullsized":"2024-07-26T06:13:37.266057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:28.831320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268031+0000","last_change":"2024-07-26T06:13:35.073326+0000","last_active":"2024-07-26T06:13:37.268031+0000","last_peered":"2024-07-26T06:13:37.268031+0000","last_clean":"2024-07-26T06:13:37.268031+0000","last_became_active":"2024-07-26T06:13:35.073019+0000","last_became_peered":"2024-07-26T06:13:35.073019+0000","last_unstale":"2024-07-26T06:13:37.268031+0000","last_undegraded":"2024-07-26T06:13:37.268031+0000","last_fullsized":"2024-07-26T06:13:37.268031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241158+0000","last_change":"2024-07-26T06:13:35.067632+0000","last_active":"2024-07-26T06:13:36.241158+0000","last_peered":"2024-07-26T06:13:36.241158+0000","last_clean":"2024-07-26T06:13:36.241158+0000","last_became_active":"2024-07-26T06:13:35.066527+0000","last_became_peered":"2024-07-26T06:13:35.066527+0000","last_unstale":"2024-07-26T06:13:36.241158+0000","last_undegraded":"2024-07-26T06:13:36.241158+0000","last_fullsized":"2024-07-26T06:13:36.241158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057001+0000","last_change":"2024-07-26T06:13:35.056477+0000","last_active":"2024-07-26T06:13:36.057001+0000","last_peered":"2024-07-26T06:13:36.057001+0000","last_clean":"2024-07-26T06:13:36.057001+0000","last_became_active":"2024-07-26T06:13:35.056222+0000","last_became_peered":"2024-07-26T06:13:35.056222+0000","last_unstale":"2024-07-26T06:13:36.057001+0000","last_undegraded":"2024-07-26T06:13:36.057001+0000","last_fullsized":"2024-07-26T06:13:36.057001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:20.776784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164819+0000","last_change":"2024-07-26T06:13:35.081680+0000","last_active":"2024-07-26T06:13:36.164819+0000","last_peered":"2024-07-26T06:13:36.164819+0000","last_clean":"2024-07-26T06:13:36.164819+0000","last_became_active":"2024-07-26T06:13:35.081409+0000","last_became_peered":"2024-07-26T06:13:35.081409+0000","last_unstale":"2024-07-26T06:13:36.164819+0000","last_undegraded":"2024-07-26T06:13:36.164819+0000","last_fullsized":"2024-07-26T06:13:36.164819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059747+0000","last_change":"2024-07-26T06:13:35.063598+0000","last_active":"2024-07-26T06:13:36.059747+0000","last_peered":"2024-07-26T06:13:36.059747+0000","last_clean":"2024-07-26T06:13:36.059747+0000","last_became_active":"2024-07-26T06:13:35.063108+0000","last_became_peered":"2024-07-26T06:13:35.063108+0000","last_unstale":"2024-07-26T06:13:36.059747+0000","last_undegraded":"2024-07-26T06:13:36.059747+0000","last_fullsized":"2024-07-26T06:13:36.059747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:49.213504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164852+0000","last_change":"2024-07-26T06:13:35.078539+0000","last_active":"2024-07-26T06:13:36.164852+0000","last_peered":"2024-07-26T06:13:36.164852+0000","last_clean":"2024-07-26T06:13:36.164852+0000","last_became_active":"2024-07-26T06:13:35.074963+0000","last_became_peered":"2024-07-26T06:13:35.074963+0000","last_unstale":"2024-07-26T06:13:36.164852+0000","last_undegraded":"2024-07-26T06:13:36.164852+0000","last_fullsized":"2024-07-26T06:13:36.164852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060557+0000","last_change":"2024-07-26T06:13:35.053573+0000","last_active":"2024-07-26T06:13:36.060557+0000","last_peered":"2024-07-26T06:13:36.060557+0000","last_clean":"2024-07-26T06:13:36.060557+0000","last_became_active":"2024-07-26T06:13:35.053330+0000","last_became_peered":"2024-07-26T06:13:35.053330+0000","last_unstale":"2024-07-26T06:13:36.060557+0000","last_undegraded":"2024-07-26T06:13:36.060557+0000","last_fullsized":"2024-07-26T06:13:36.060557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:01.836541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241629+0000","last_change":"2024-07-26T06:13:35.074100+0000","last_active":"2024-07-26T06:13:36.241629+0000","last_peered":"2024-07-26T06:13:36.241629+0000","last_clean":"2024-07-26T06:13:36.241629+0000","last_became_active":"2024-07-26T06:13:35.073532+0000","last_became_peered":"2024-07-26T06:13:35.073532+0000","last_unstale":"2024-07-26T06:13:36.241629+0000","last_undegraded":"2024-07-26T06:13:36.241629+0000","last_fullsized":"2024-07-26T06:13:36.241629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059866+0000","last_change":"2024-07-26T06:13:35.053991+0000","last_active":"2024-07-26T06:13:35.059866+0000","last_peered":"2024-07-26T06:13:35.059866+0000","last_clean":"2024-07-26T06:13:35.059866+0000","last_became_active":"2024-07-26T06:13:35.053723+0000","last_became_peered":"2024-07-26T06:13:35.053723+0000","last_unstale":"2024-07-26T06:13:35.059866+0000","last_undegraded":"2024-07-26T06:13:35.059866+0000","last_fullsized":"2024-07-26T06:13:35.059866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:38.085872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060277+0000","last_change":"2024-07-26T06:13:35.055687+0000","last_active":"2024-07-26T06:13:36.060277+0000","last_peered":"2024-07-26T06:13:36.060277+0000","last_clean":"2024-07-26T06:13:36.060277+0000","last_became_active":"2024-07-26T06:13:35.054863+0000","last_became_peered":"2024-07-26T06:13:35.054863+0000","last_unstale":"2024-07-26T06:13:36.060277+0000","last_undegraded":"2024-07-26T06:13:36.060277+0000","last_fullsized":"2024-07-26T06:13:36.060277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.827844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059098+0000","last_change":"2024-07-26T06:13:35.047582+0000","last_active":"2024-07-26T06:13:36.059098+0000","last_peered":"2024-07-26T06:13:36.059098+0000","last_clean":"2024-07-26T06:13:36.059098+0000","last_became_active":"2024-07-26T06:13:35.047322+0000","last_became_peered":"2024-07-26T06:13:35.047322+0000","last_unstale":"2024-07-26T06:13:36.059098+0000","last_undegraded":"2024-07-26T06:13:36.059098+0000","last_fullsized":"2024-07-26T06:13:36.059098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165246+0000","last_change":"2024-07-26T06:13:35.079808+0000","last_active":"2024-07-26T06:13:36.165246+0000","last_peered":"2024-07-26T06:13:36.165246+0000","last_clean":"2024-07-26T06:13:36.165246+0000","last_became_active":"2024-07-26T06:13:35.076583+0000","last_became_peered":"2024-07-26T06:13:35.076583+0000","last_unstale":"2024-07-26T06:13:36.165246+0000","last_undegraded":"2024-07-26T06:13:36.165246+0000","last_fullsized":"2024-07-26T06:13:36.165246+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265681+0000","last_change":"2024-07-26T06:13:35.062887+0000","last_active":"2024-07-26T06:13:37.265681+0000","last_peered":"2024-07-26T06:13:37.265681+0000","last_clean":"2024-07-26T06:13:37.265681+0000","last_became_active":"2024-07-26T06:13:35.062553+0000","last_became_peered":"2024-07-26T06:13:35.062553+0000","last_unstale":"2024-07-26T06:13:37.265681+0000","last_undegraded":"2024-07-26T06:13:37.265681+0000","last_fullsized":"2024-07-26T06:13:37.265681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:38.675161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240837+0000","last_change":"2024-07-26T06:13:35.059916+0000","last_active":"2024-07-26T06:13:36.240837+0000","last_peered":"2024-07-26T06:13:36.240837+0000","last_clean":"2024-07-26T06:13:36.240837+0000","last_became_active":"2024-07-26T06:13:35.058276+0000","last_became_peered":"2024-07-26T06:13:35.058276+0000","last_unstale":"2024-07-26T06:13:36.240837+0000","last_undegraded":"2024-07-26T06:13:36.240837+0000","last_fullsized":"2024-07-26T06:13:36.240837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062155+0000","last_change":"2024-07-26T06:13:35.052170+0000","last_active":"2024-07-26T06:13:36.062155+0000","last_peered":"2024-07-26T06:13:36.062155+0000","last_clean":"2024-07-26T06:13:36.062155+0000","last_became_active":"2024-07-26T06:13:35.051630+0000","last_became_peered":"2024-07-26T06:13:35.051630+0000","last_unstale":"2024-07-26T06:13:36.062155+0000","last_undegraded":"2024-07-26T06:13:36.062155+0000","last_fullsized":"2024-07-26T06:13:36.062155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:31.103018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059899+0000","last_change":"2024-07-26T06:13:34.065590+0000","last_active":"2024-07-26T06:13:36.059899+0000","last_peered":"2024-07-26T06:13:36.059899+0000","last_clean":"2024-07-26T06:13:36.059899+0000","last_became_active":"2024-07-26T06:13:34.065275+0000","last_became_peered":"2024-07-26T06:13:34.065275+0000","last_unstale":"2024-07-26T06:13:36.059899+0000","last_undegraded":"2024-07-26T06:13:36.059899+0000","last_fullsized":"2024-07-26T06:13:36.059899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059022+0000","last_change":"2024-07-26T06:13:35.057161+0000","last_active":"2024-07-26T06:13:36.059022+0000","last_peered":"2024-07-26T06:13:36.059022+0000","last_clean":"2024-07-26T06:13:36.059022+0000","last_became_active":"2024-07-26T06:13:35.056843+0000","last_became_peered":"2024-07-26T06:13:35.056843+0000","last_unstale":"2024-07-26T06:13:36.059022+0000","last_undegraded":"2024-07-26T06:13:36.059022+0000","last_fullsized":"2024-07-26T06:13:36.059022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241143+0000","last_change":"2024-07-26T06:13:34.061026+0000","last_active":"2024-07-26T06:13:36.241143+0000","last_peered":"2024-07-26T06:13:36.241143+0000","last_clean":"2024-07-26T06:13:36.241143+0000","last_became_active":"2024-07-26T06:13:34.060729+0000","last_became_peered":"2024-07-26T06:13:34.060729+0000","last_unstale":"2024-07-26T06:13:36.241143+0000","last_undegraded":"2024-07-26T06:13:36.241143+0000","last_fullsized":"2024-07-26T06:13:36.241143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059737+0000","last_change":"2024-07-26T06:13:34.053732+0000","last_active":"2024-07-26T06:13:36.059737+0000","last_peered":"2024-07-26T06:13:36.059737+0000","last_clean":"2024-07-26T06:13:36.059737+0000","last_became_active":"2024-07-26T06:13:34.053385+0000","last_became_peered":"2024-07-26T06:13:34.053385+0000","last_unstale":"2024-07-26T06:13:36.059737+0000","last_undegraded":"2024-07-26T06:13:36.059737+0000","last_fullsized":"2024-07-26T06:13:36.059737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:24:03.458270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059243+0000","last_change":"2024-07-26T06:13:35.042064+0000","last_active":"2024-07-26T06:13:36.059243+0000","last_peered":"2024-07-26T06:13:36.059243+0000","last_clean":"2024-07-26T06:13:36.059243+0000","last_became_active":"2024-07-26T06:13:35.041741+0000","last_became_peered":"2024-07-26T06:13:35.041741+0000","last_unstale":"2024-07-26T06:13:36.059243+0000","last_undegraded":"2024-07-26T06:13:36.059243+0000","last_fullsized":"2024-07-26T06:13:36.059243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:20.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165112+0000","last_change":"2024-07-26T06:13:34.061502+0000","last_active":"2024-07-26T06:13:36.165112+0000","last_peered":"2024-07-26T06:13:36.165112+0000","last_clean":"2024-07-26T06:13:36.165112+0000","last_became_active":"2024-07-26T06:13:34.061155+0000","last_became_peered":"2024-07-26T06:13:34.061155+0000","last_unstale":"2024-07-26T06:13:36.165112+0000","last_undegraded":"2024-07-26T06:13:36.165112+0000","last_fullsized":"2024-07-26T06:13:36.165112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059242+0000","last_change":"2024-07-26T06:13:35.051729+0000","last_active":"2024-07-26T06:13:35.059242+0000","last_peered":"2024-07-26T06:13:35.059242+0000","last_clean":"2024-07-26T06:13:35.059242+0000","last_became_active":"2024-07-26T06:13:35.051429+0000","last_became_peered":"2024-07-26T06:13:35.051429+0000","last_unstale":"2024-07-26T06:13:35.059242+0000","last_undegraded":"2024-07-26T06:13:35.059242+0000","last_fullsized":"2024-07-26T06:13:35.059242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:15:06.230764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241239+0000","last_change":"2024-07-26T06:13:34.066637+0000","last_active":"2024-07-26T06:13:36.241239+0000","last_peered":"2024-07-26T06:13:36.241239+0000","last_clean":"2024-07-26T06:13:36.241239+0000","last_became_active":"2024-07-26T06:13:34.066225+0000","last_became_peered":"2024-07-26T06:13:34.066225+0000","last_unstale":"2024-07-26T06:13:36.241239+0000","last_undegraded":"2024-07-26T06:13:36.241239+0000","last_fullsized":"2024-07-26T06:13:36.241239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165378+0000","last_change":"2024-07-26T06:13:35.080900+0000","last_active":"2024-07-26T06:13:36.165378+0000","last_peered":"2024-07-26T06:13:36.165378+0000","last_clean":"2024-07-26T06:13:36.165378+0000","last_became_active":"2024-07-26T06:13:35.080442+0000","last_became_peered":"2024-07-26T06:13:35.080442+0000","last_unstale":"2024-07-26T06:13:36.165378+0000","last_undegraded":"2024-07-26T06:13:36.165378+0000","last_fullsized":"2024-07-26T06:13:36.165378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165433+0000","last_change":"2024-07-26T06:13:34.060576+0000","last_active":"2024-07-26T06:13:36.165433+0000","last_peered":"2024-07-26T06:13:36.165433+0000","last_clean":"2024-07-26T06:13:36.165433+0000","last_became_active":"2024-07-26T06:13:34.060215+0000","last_became_peered":"2024-07-26T06:13:34.060215+0000","last_unstale":"2024-07-26T06:13:36.165433+0000","last_undegraded":"2024-07-26T06:13:36.165433+0000","last_fullsized":"2024-07-26T06:13:36.165433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:19.760539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057253+0000","last_change":"2024-07-26T06:13:35.072548+0000","last_active":"2024-07-26T06:13:36.057253+0000","last_peered":"2024-07-26T06:13:36.057253+0000","last_clean":"2024-07-26T06:13:36.057253+0000","last_became_active":"2024-07-26T06:13:35.072227+0000","last_became_peered":"2024-07-26T06:13:35.072227+0000","last_unstale":"2024-07-26T06:13:36.057253+0000","last_undegraded":"2024-07-26T06:13:36.057253+0000","last_fullsized":"2024-07-26T06:13:36.057253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.081899+0000","last_change":"2024-07-26T06:13:34.064165+0000","last_active":"2024-07-26T06:13:35.081899+0000","last_peered":"2024-07-26T06:13:35.081899+0000","last_clean":"2024-07-26T06:13:35.081899+0000","last_became_active":"2024-07-26T06:13:34.063828+0000","last_became_peered":"2024-07-26T06:13:34.063828+0000","last_unstale":"2024-07-26T06:13:35.081899+0000","last_undegraded":"2024-07-26T06:13:35.081899+0000","last_fullsized":"2024-07-26T06:13:35.081899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:30.086773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063050+0000","last_change":"2024-07-26T06:13:35.064262+0000","last_active":"2024-07-26T06:13:36.063050+0000","last_peered":"2024-07-26T06:13:36.063050+0000","last_clean":"2024-07-26T06:13:36.063050+0000","last_became_active":"2024-07-26T06:13:35.061569+0000","last_became_peered":"2024-07-26T06:13:35.061569+0000","last_unstale":"2024-07-26T06:13:36.063050+0000","last_undegraded":"2024-07-26T06:13:36.063050+0000","last_fullsized":"2024-07-26T06:13:36.063050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:15:06.230764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241453+0000","last_change":"2024-07-26T06:13:34.053539+0000","last_active":"2024-07-26T06:13:36.241453+0000","last_peered":"2024-07-26T06:13:36.241453+0000","last_clean":"2024-07-26T06:13:36.241453+0000","last_became_active":"2024-07-26T06:13:34.053038+0000","last_became_peered":"2024-07-26T06:13:34.053038+0000","last_unstale":"2024-07-26T06:13:36.241453+0000","last_undegraded":"2024-07-26T06:13:36.241453+0000","last_fullsized":"2024-07-26T06:13:36.241453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267003+0000","last_change":"2024-07-26T06:13:35.075023+0000","last_active":"2024-07-26T06:13:37.267003+0000","last_peered":"2024-07-26T06:13:37.267003+0000","last_clean":"2024-07-26T06:13:37.267003+0000","last_became_active":"2024-07-26T06:13:35.073931+0000","last_became_peered":"2024-07-26T06:13:35.073931+0000","last_unstale":"2024-07-26T06:13:37.267003+0000","last_undegraded":"2024-07-26T06:13:37.267003+0000","last_fullsized":"2024-07-26T06:13:37.267003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059213+0000","last_change":"2024-07-26T06:13:34.048636+0000","last_active":"2024-07-26T06:13:36.059213+0000","last_peered":"2024-07-26T06:13:36.059213+0000","last_clean":"2024-07-26T06:13:36.059213+0000","last_became_active":"2024-07-26T06:13:34.048001+0000","last_became_peered":"2024-07-26T06:13:34.048001+0000","last_unstale":"2024-07-26T06:13:36.059213+0000","last_undegraded":"2024-07-26T06:13:36.059213+0000","last_fullsized":"2024-07-26T06:13:36.059213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:11:14.256545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.058850+0000","last_change":"2024-07-26T06:13:35.057953+0000","last_active":"2024-07-26T06:13:36.058850+0000","last_peered":"2024-07-26T06:13:36.058850+0000","last_clean":"2024-07-26T06:13:36.058850+0000","last_became_active":"2024-07-26T06:13:35.057762+0000","last_became_peered":"2024-07-26T06:13:35.057762+0000","last_unstale":"2024-07-26T06:13:36.058850+0000","last_undegraded":"2024-07-26T06:13:36.058850+0000","last_fullsized":"2024-07-26T06:13:36.058850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165279+0000","last_change":"2024-07-26T06:13:34.058750+0000","last_active":"2024-07-26T06:13:36.165279+0000","last_peered":"2024-07-26T06:13:36.165279+0000","last_clean":"2024-07-26T06:13:36.165279+0000","last_became_active":"2024-07-26T06:13:34.057317+0000","last_became_peered":"2024-07-26T06:13:34.057317+0000","last_unstale":"2024-07-26T06:13:36.165279+0000","last_undegraded":"2024-07-26T06:13:36.165279+0000","last_fullsized":"2024-07-26T06:13:36.165279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267198+0000","last_change":"2024-07-26T06:13:35.068556+0000","last_active":"2024-07-26T06:13:37.267198+0000","last_peered":"2024-07-26T06:13:37.267198+0000","last_clean":"2024-07-26T06:13:37.267198+0000","last_became_active":"2024-07-26T06:13:35.067965+0000","last_became_peered":"2024-07-26T06:13:35.067965+0000","last_unstale":"2024-07-26T06:13:37.267198+0000","last_undegraded":"2024-07-26T06:13:37.267198+0000","last_fullsized":"2024-07-26T06:13:37.267198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241420+0000","last_change":"2024-07-26T06:13:34.062439+0000","last_active":"2024-07-26T06:13:36.241420+0000","last_peered":"2024-07-26T06:13:36.241420+0000","last_clean":"2024-07-26T06:13:36.241420+0000","last_became_active":"2024-07-26T06:13:34.061547+0000","last_became_peered":"2024-07-26T06:13:34.061547+0000","last_unstale":"2024-07-26T06:13:36.241420+0000","last_undegraded":"2024-07-26T06:13:36.241420+0000","last_fullsized":"2024-07-26T06:13:36.241420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240885+0000","last_change":"2024-07-26T06:13:35.059120+0000","last_active":"2024-07-26T06:13:36.240885+0000","last_peered":"2024-07-26T06:13:36.240885+0000","last_clean":"2024-07-26T06:13:36.240885+0000","last_became_active":"2024-07-26T06:13:35.058851+0000","last_became_peered":"2024-07-26T06:13:35.058851+0000","last_unstale":"2024-07-26T06:13:36.240885+0000","last_undegraded":"2024-07-26T06:13:36.240885+0000","last_fullsized":"2024-07-26T06:13:36.240885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265520+0000","last_change":"2024-07-26T06:13:34.052072+0000","last_active":"2024-07-26T06:13:37.265520+0000","last_peered":"2024-07-26T06:13:37.265520+0000","last_clean":"2024-07-26T06:13:37.265520+0000","last_became_active":"2024-07-26T06:13:34.051775+0000","last_became_peered":"2024-07-26T06:13:34.051775+0000","last_unstale":"2024-07-26T06:13:37.265520+0000","last_undegraded":"2024-07-26T06:13:37.265520+0000","last_fullsized":"2024-07-26T06:13:37.265520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266571+0000","last_change":"2024-07-26T06:13:35.069871+0000","last_active":"2024-07-26T06:13:37.266571+0000","last_peered":"2024-07-26T06:13:37.266571+0000","last_clean":"2024-07-26T06:13:37.266571+0000","last_became_active":"2024-07-26T06:13:35.069272+0000","last_became_peered":"2024-07-26T06:13:35.069272+0000","last_unstale":"2024-07-26T06:13:37.266571+0000","last_undegraded":"2024-07-26T06:13:37.266571+0000","last_fullsized":"2024-07-26T06:13:37.266571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062736+0000","last_change":"2024-07-26T06:13:34.050786+0000","last_active":"2024-07-26T06:13:36.062736+0000","last_peered":"2024-07-26T06:13:36.062736+0000","last_clean":"2024-07-26T06:13:36.062736+0000","last_became_active":"2024-07-26T06:13:34.050291+0000","last_became_peered":"2024-07-26T06:13:34.050291+0000","last_unstale":"2024-07-26T06:13:36.062736+0000","last_undegraded":"2024-07-26T06:13:36.062736+0000","last_fullsized":"2024-07-26T06:13:36.062736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:28:47.174016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267602+0000","last_change":"2024-07-26T06:13:35.077176+0000","last_active":"2024-07-26T06:13:37.267602+0000","last_peered":"2024-07-26T06:13:37.267602+0000","last_clean":"2024-07-26T06:13:37.267602+0000","last_became_active":"2024-07-26T06:13:35.074703+0000","last_became_peered":"2024-07-26T06:13:35.074703+0000","last_unstale":"2024-07-26T06:13:37.267602+0000","last_undegraded":"2024-07-26T06:13:37.267602+0000","last_fullsized":"2024-07-26T06:13:37.267602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266222+0000","last_change":"2024-07-26T06:13:34.052622+0000","last_active":"2024-07-26T06:13:37.266222+0000","last_peered":"2024-07-26T06:13:37.266222+0000","last_clean":"2024-07-26T06:13:37.266222+0000","last_became_active":"2024-07-26T06:13:34.052311+0000","last_became_peered":"2024-07-26T06:13:34.052311+0000","last_unstale":"2024-07-26T06:13:37.266222+0000","last_undegraded":"2024-07-26T06:13:37.266222+0000","last_fullsized":"2024-07-26T06:13:37.266222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268227+0000","last_change":"2024-07-26T06:13:35.078765+0000","last_active":"2024-07-26T06:13:37.268227+0000","last_peered":"2024-07-26T06:13:37.268227+0000","last_clean":"2024-07-26T06:13:37.268227+0000","last_became_active":"2024-07-26T06:13:35.077968+0000","last_became_peered":"2024-07-26T06:13:35.077968+0000","last_unstale":"2024-07-26T06:13:37.268227+0000","last_undegraded":"2024-07-26T06:13:37.268227+0000","last_fullsized":"2024-07-26T06:13:37.268227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:22.947178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.067050+0000","last_change":"2024-07-26T06:13:34.047184+0000","last_active":"2024-07-26T06:13:35.067050+0000","last_peered":"2024-07-26T06:13:35.067050+0000","last_clean":"2024-07-26T06:13:35.067050+0000","last_became_active":"2024-07-26T06:13:34.046868+0000","last_became_peered":"2024-07-26T06:13:34.046868+0000","last_unstale":"2024-07-26T06:13:35.067050+0000","last_undegraded":"2024-07-26T06:13:35.067050+0000","last_fullsized":"2024-07-26T06:13:35.067050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:17.313633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165826+0000","last_change":"2024-07-26T06:13:35.065849+0000","last_active":"2024-07-26T06:13:36.165826+0000","last_peered":"2024-07-26T06:13:36.165826+0000","last_clean":"2024-07-26T06:13:36.165826+0000","last_became_active":"2024-07-26T06:13:35.065135+0000","last_became_peered":"2024-07-26T06:13:35.065135+0000","last_unstale":"2024-07-26T06:13:36.165826+0000","last_undegraded":"2024-07-26T06:13:36.165826+0000","last_fullsized":"2024-07-26T06:13:36.165826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265761+0000","last_change":"2024-07-26T06:13:34.062785+0000","last_active":"2024-07-26T06:13:37.265761+0000","last_peered":"2024-07-26T06:13:37.265761+0000","last_clean":"2024-07-26T06:13:37.265761+0000","last_became_active":"2024-07-26T06:13:34.062466+0000","last_became_peered":"2024-07-26T06:13:34.062466+0000","last_unstale":"2024-07-26T06:13:37.265761+0000","last_undegraded":"2024-07-26T06:13:37.265761+0000","last_fullsized":"2024-07-26T06:13:37.265761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:13:51.856008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267453+0000","last_change":"2024-07-26T06:13:35.082619+0000","last_active":"2024-07-26T06:13:37.267453+0000","last_peered":"2024-07-26T06:13:37.267453+0000","last_clean":"2024-07-26T06:13:37.267453+0000","last_became_active":"2024-07-26T06:13:35.082044+0000","last_became_peered":"2024-07-26T06:13:35.082044+0000","last_unstale":"2024-07-26T06:13:37.267453+0000","last_undegraded":"2024-07-26T06:13:37.267453+0000","last_fullsized":"2024-07-26T06:13:37.267453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266471+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:13:37.266471+0000","last_peered":"2024-07-26T06:13:37.266471+0000","last_clean":"2024-07-26T06:13:37.266471+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:13:37.266471+0000","last_undegraded":"2024-07-26T06:13:37.266471+0000","last_fullsized":"2024-07-26T06:13:37.266471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164685+0000","last_change":"2024-07-26T06:13:34.061955+0000","last_active":"2024-07-26T06:13:36.164685+0000","last_peered":"2024-07-26T06:13:36.164685+0000","last_clean":"2024-07-26T06:13:36.164685+0000","last_became_active":"2024-07-26T06:13:34.060172+0000","last_became_peered":"2024-07-26T06:13:34.060172+0000","last_unstale":"2024-07-26T06:13:36.164685+0000","last_undegraded":"2024-07-26T06:13:36.164685+0000","last_fullsized":"2024-07-26T06:13:36.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:45.461779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059576+0000","last_change":"2024-07-26T06:13:35.077259+0000","last_active":"2024-07-26T06:13:36.059576+0000","last_peered":"2024-07-26T06:13:36.059576+0000","last_clean":"2024-07-26T06:13:36.059576+0000","last_became_active":"2024-07-26T06:13:35.076879+0000","last_became_peered":"2024-07-26T06:13:35.076879+0000","last_unstale":"2024-07-26T06:13:36.059576+0000","last_undegraded":"2024-07-26T06:13:36.059576+0000","last_fullsized":"2024-07-26T06:13:36.059576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:27.327738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266356+0000","last_change":"2024-07-26T06:13:34.064601+0000","last_active":"2024-07-26T06:13:37.266356+0000","last_peered":"2024-07-26T06:13:37.266356+0000","last_clean":"2024-07-26T06:13:37.266356+0000","last_became_active":"2024-07-26T06:13:34.064270+0000","last_became_peered":"2024-07-26T06:13:34.064270+0000","last_unstale":"2024-07-26T06:13:37.266356+0000","last_undegraded":"2024-07-26T06:13:37.266356+0000","last_fullsized":"2024-07-26T06:13:37.266356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056647+0000","last_change":"2024-07-26T06:13:35.062026+0000","last_active":"2024-07-26T06:13:36.056647+0000","last_peered":"2024-07-26T06:13:36.056647+0000","last_clean":"2024-07-26T06:13:36.056647+0000","last_became_active":"2024-07-26T06:13:35.061732+0000","last_became_peered":"2024-07-26T06:13:35.061732+0000","last_unstale":"2024-07-26T06:13:36.056647+0000","last_undegraded":"2024-07-26T06:13:36.056647+0000","last_fullsized":"2024-07-26T06:13:36.056647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:32.385097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265755+0000","last_change":"2024-07-26T06:13:34.057752+0000","last_active":"2024-07-26T06:13:37.265755+0000","last_peered":"2024-07-26T06:13:37.265755+0000","last_clean":"2024-07-26T06:13:37.265755+0000","last_became_active":"2024-07-26T06:13:34.057421+0000","last_became_peered":"2024-07-26T06:13:34.057421+0000","last_unstale":"2024-07-26T06:13:37.265755+0000","last_undegraded":"2024-07-26T06:13:37.265755+0000","last_fullsized":"2024-07-26T06:13:37.265755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059275+0000","last_change":"2024-07-26T06:13:35.051454+0000","last_active":"2024-07-26T06:13:35.059275+0000","last_peered":"2024-07-26T06:13:35.059275+0000","last_clean":"2024-07-26T06:13:35.059275+0000","last_became_active":"2024-07-26T06:13:35.051175+0000","last_became_peered":"2024-07-26T06:13:35.051175+0000","last_unstale":"2024-07-26T06:13:35.059275+0000","last_undegraded":"2024-07-26T06:13:35.059275+0000","last_fullsized":"2024-07-26T06:13:35.059275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:50:23.338780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165555+0000","last_change":"2024-07-26T06:13:34.055084+0000","last_active":"2024-07-26T06:13:36.165555+0000","last_peered":"2024-07-26T06:13:36.165555+0000","last_clean":"2024-07-26T06:13:36.165555+0000","last_became_active":"2024-07-26T06:13:34.054390+0000","last_became_peered":"2024-07-26T06:13:34.054390+0000","last_unstale":"2024-07-26T06:13:36.165555+0000","last_undegraded":"2024-07-26T06:13:36.165555+0000","last_fullsized":"2024-07-26T06:13:36.165555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062792+0000","last_change":"2024-07-26T06:13:35.052783+0000","last_active":"2024-07-26T06:13:36.062792+0000","last_peered":"2024-07-26T06:13:36.062792+0000","last_clean":"2024-07-26T06:13:36.062792+0000","last_became_active":"2024-07-26T06:13:35.048901+0000","last_became_peered":"2024-07-26T06:13:35.048901+0000","last_unstale":"2024-07-26T06:13:36.062792+0000","last_undegraded":"2024-07-26T06:13:36.062792+0000","last_fullsized":"2024-07-26T06:13:36.062792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:18.329878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.049721+0000","last_change":"2024-07-26T06:13:34.049038+0000","last_active":"2024-07-26T06:13:35.049721+0000","last_peered":"2024-07-26T06:13:35.049721+0000","last_clean":"2024-07-26T06:13:35.049721+0000","last_became_active":"2024-07-26T06:13:34.048751+0000","last_became_peered":"2024-07-26T06:13:34.048751+0000","last_unstale":"2024-07-26T06:13:35.049721+0000","last_undegraded":"2024-07-26T06:13:35.049721+0000","last_fullsized":"2024-07-26T06:13:35.049721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165618+0000","last_change":"2024-07-26T06:13:35.080209+0000","last_active":"2024-07-26T06:13:36.165618+0000","last_peered":"2024-07-26T06:13:36.165618+0000","last_clean":"2024-07-26T06:13:36.165618+0000","last_became_active":"2024-07-26T06:13:35.079870+0000","last_became_peered":"2024-07-26T06:13:35.079870+0000","last_unstale":"2024-07-26T06:13:36.165618+0000","last_undegraded":"2024-07-26T06:13:36.165618+0000","last_fullsized":"2024-07-26T06:13:36.165618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268083+0000","last_change":"2024-07-26T06:13:34.066620+0000","last_active":"2024-07-26T06:13:37.268083+0000","last_peered":"2024-07-26T06:13:37.268083+0000","last_clean":"2024-07-26T06:13:37.268083+0000","last_became_active":"2024-07-26T06:13:34.066296+0000","last_became_peered":"2024-07-26T06:13:34.066296+0000","last_unstale":"2024-07-26T06:13:37.268083+0000","last_undegraded":"2024-07-26T06:13:37.268083+0000","last_fullsized":"2024-07-26T06:13:37.268083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:09:26.010192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.059439+0000","last_change":"2024-07-26T06:13:35.052176+0000","last_active":"2024-07-26T06:13:35.059439+0000","last_peered":"2024-07-26T06:13:35.059439+0000","last_clean":"2024-07-26T06:13:35.059439+0000","last_became_active":"2024-07-26T06:13:35.051905+0000","last_became_peered":"2024-07-26T06:13:35.051905+0000","last_unstale":"2024-07-26T06:13:35.059439+0000","last_undegraded":"2024-07-26T06:13:35.059439+0000","last_fullsized":"2024-07-26T06:13:35.059439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:41.825450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.049853+0000","last_change":"2024-07-26T06:13:34.059549+0000","last_active":"2024-07-26T06:13:35.049853+0000","last_peered":"2024-07-26T06:13:35.049853+0000","last_clean":"2024-07-26T06:13:35.049853+0000","last_became_active":"2024-07-26T06:13:34.058721+0000","last_became_peered":"2024-07-26T06:13:34.058721+0000","last_unstale":"2024-07-26T06:13:35.049853+0000","last_undegraded":"2024-07-26T06:13:35.049853+0000","last_fullsized":"2024-07-26T06:13:35.049853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060701+0000","last_change":"2024-07-26T06:13:35.066322+0000","last_active":"2024-07-26T06:13:36.060701+0000","last_peered":"2024-07-26T06:13:36.060701+0000","last_clean":"2024-07-26T06:13:36.060701+0000","last_became_active":"2024-07-26T06:13:35.066006+0000","last_became_peered":"2024-07-26T06:13:35.066006+0000","last_unstale":"2024-07-26T06:13:36.060701+0000","last_undegraded":"2024-07-26T06:13:36.060701+0000","last_fullsized":"2024-07-26T06:13:36.060701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:31.900907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241473+0000","last_change":"2024-07-26T06:13:34.069982+0000","last_active":"2024-07-26T06:13:36.241473+0000","last_peered":"2024-07-26T06:13:36.241473+0000","last_clean":"2024-07-26T06:13:36.241473+0000","last_became_active":"2024-07-26T06:13:34.069000+0000","last_became_peered":"2024-07-26T06:13:34.069000+0000","last_unstale":"2024-07-26T06:13:36.241473+0000","last_undegraded":"2024-07-26T06:13:36.241473+0000","last_fullsized":"2024-07-26T06:13:36.241473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059318+0000","last_change":"2024-07-26T06:13:35.059073+0000","last_active":"2024-07-26T06:13:36.059318+0000","last_peered":"2024-07-26T06:13:36.059318+0000","last_clean":"2024-07-26T06:13:36.059318+0000","last_became_active":"2024-07-26T06:13:35.058724+0000","last_became_peered":"2024-07-26T06:13:35.058724+0000","last_unstale":"2024-07-26T06:13:36.059318+0000","last_undegraded":"2024-07-26T06:13:36.059318+0000","last_fullsized":"2024-07-26T06:13:36.059318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266310+0000","last_change":"2024-07-26T06:13:34.065271+0000","last_active":"2024-07-26T06:13:37.266310+0000","last_peered":"2024-07-26T06:13:37.266310+0000","last_clean":"2024-07-26T06:13:37.266310+0000","last_became_active":"2024-07-26T06:13:34.064922+0000","last_became_peered":"2024-07-26T06:13:34.064922+0000","last_unstale":"2024-07-26T06:13:37.266310+0000","last_undegraded":"2024-07-26T06:13:37.266310+0000","last_fullsized":"2024-07-26T06:13:37.266310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062133+0000","last_change":"2024-07-26T06:13:35.056227+0000","last_active":"2024-07-26T06:13:36.062133+0000","last_peered":"2024-07-26T06:13:36.062133+0000","last_clean":"2024-07-26T06:13:36.062133+0000","last_became_active":"2024-07-26T06:13:35.055855+0000","last_became_peered":"2024-07-26T06:13:35.055855+0000","last_unstale":"2024-07-26T06:13:36.062133+0000","last_undegraded":"2024-07-26T06:13:36.062133+0000","last_fullsized":"2024-07-26T06:13:36.062133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:03.118475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.050762+0000","last_change":"2024-07-26T06:13:34.066397+0000","last_active":"2024-07-26T06:13:35.050762+0000","last_peered":"2024-07-26T06:13:35.050762+0000","last_clean":"2024-07-26T06:13:35.050762+0000","last_became_active":"2024-07-26T06:13:34.066084+0000","last_became_peered":"2024-07-26T06:13:34.066084+0000","last_unstale":"2024-07-26T06:13:35.050762+0000","last_undegraded":"2024-07-26T06:13:35.050762+0000","last_fullsized":"2024-07-26T06:13:35.050762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:02.102230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059415+0000","last_change":"2024-07-26T06:13:35.053587+0000","last_active":"2024-07-26T06:13:36.059415+0000","last_peered":"2024-07-26T06:13:36.059415+0000","last_clean":"2024-07-26T06:13:36.059415+0000","last_became_active":"2024-07-26T06:13:35.053237+0000","last_became_peered":"2024-07-26T06:13:35.053237+0000","last_unstale":"2024-07-26T06:13:36.059415+0000","last_undegraded":"2024-07-26T06:13:36.059415+0000","last_fullsized":"2024-07-26T06:13:36.059415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:07.936415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-07-26T06:13:35.050114+0000","last_change":"2024-07-26T06:13:34.044567+0000","last_active":"2024-07-26T06:13:35.050114+0000","last_peered":"2024-07-26T06:13:35.050114+0000","last_clean":"2024-07-26T06:13:35.050114+0000","last_became_active":"2024-07-26T06:13:34.043889+0000","last_became_peered":"2024-07-26T06:13:34.043889+0000","last_unstale":"2024-07-26T06:13:35.050114+0000","last_undegraded":"2024-07-26T06:13:35.050114+0000","last_fullsized":"2024-07-26T06:13:35.050114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:40.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060003+0000","last_change":"2024-07-26T06:13:35.043595+0000","last_active":"2024-07-26T06:13:36.060003+0000","last_peered":"2024-07-26T06:13:36.060003+0000","last_clean":"2024-07-26T06:13:36.060003+0000","last_became_active":"2024-07-26T06:13:35.043258+0000","last_became_peered":"2024-07-26T06:13:35.043258+0000","last_unstale":"2024-07-26T06:13:36.060003+0000","last_undegraded":"2024-07-26T06:13:36.060003+0000","last_fullsized":"2024-07-26T06:13:36.060003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:48.040021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267509+0000","last_change":"2024-07-26T06:13:34.064636+0000","last_active":"2024-07-26T06:13:37.267509+0000","last_peered":"2024-07-26T06:13:37.267509+0000","last_clean":"2024-07-26T06:13:37.267509+0000","last_became_active":"2024-07-26T06:13:34.064308+0000","last_became_peered":"2024-07-26T06:13:34.064308+0000","last_unstale":"2024-07-26T06:13:37.267509+0000","last_undegraded":"2024-07-26T06:13:37.267509+0000","last_fullsized":"2024-07-26T06:13:37.267509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240857+0000","last_change":"2024-07-26T06:13:35.066830+0000","last_active":"2024-07-26T06:13:36.240857+0000","last_peered":"2024-07-26T06:13:36.240857+0000","last_clean":"2024-07-26T06:13:36.240857+0000","last_became_active":"2024-07-26T06:13:35.066494+0000","last_became_peered":"2024-07-26T06:13:35.066494+0000","last_unstale":"2024-07-26T06:13:36.240857+0000","last_undegraded":"2024-07-26T06:13:36.240857+0000","last_fullsized":"2024-07-26T06:13:36.240857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165479+0000","last_change":"2024-07-26T06:13:35.079246+0000","last_active":"2024-07-26T06:13:36.165479+0000","last_peered":"2024-07-26T06:13:36.165479+0000","last_clean":"2024-07-26T06:13:36.165479+0000","last_became_active":"2024-07-26T06:13:35.075982+0000","last_became_peered":"2024-07-26T06:13:35.075982+0000","last_unstale":"2024-07-26T06:13:36.165479+0000","last_undegraded":"2024-07-26T06:13:36.165479+0000","last_fullsized":"2024-07-26T06:13:36.165479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267123+0000","last_change":"2024-07-26T06:13:34.066151+0000","last_active":"2024-07-26T06:13:37.267123+0000","last_peered":"2024-07-26T06:13:37.267123+0000","last_clean":"2024-07-26T06:13:37.267123+0000","last_became_active":"2024-07-26T06:13:34.065737+0000","last_became_peered":"2024-07-26T06:13:34.065737+0000","last_unstale":"2024-07-26T06:13:37.267123+0000","last_undegraded":"2024-07-26T06:13:37.267123+0000","last_fullsized":"2024-07-26T06:13:37.267123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165512+0000","last_change":"2024-07-26T06:13:35.075547+0000","last_active":"2024-07-26T06:13:36.165512+0000","last_peered":"2024-07-26T06:13:36.165512+0000","last_clean":"2024-07-26T06:13:36.165512+0000","last_became_active":"2024-07-26T06:13:35.075262+0000","last_became_peered":"2024-07-26T06:13:35.075262+0000","last_unstale":"2024-07-26T06:13:36.165512+0000","last_undegraded":"2024-07-26T06:13:36.165512+0000","last_fullsized":"2024-07-26T06:13:36.165512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267491+0000","last_change":"2024-07-26T06:13:34.066478+0000","last_active":"2024-07-26T06:13:37.267491+0000","last_peered":"2024-07-26T06:13:37.267491+0000","last_clean":"2024-07-26T06:13:37.267491+0000","last_became_active":"2024-07-26T06:13:34.066015+0000","last_became_peered":"2024-07-26T06:13:34.066015+0000","last_unstale":"2024-07-26T06:13:37.267491+0000","last_undegraded":"2024-07-26T06:13:37.267491+0000","last_fullsized":"2024-07-26T06:13:37.267491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:27.815076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059941+0000","last_change":"2024-07-26T06:13:35.077607+0000","last_active":"2024-07-26T06:13:36.059941+0000","last_peered":"2024-07-26T06:13:36.059941+0000","last_clean":"2024-07-26T06:13:36.059941+0000","last_became_active":"2024-07-26T06:13:35.077423+0000","last_became_peered":"2024-07-26T06:13:35.077423+0000","last_unstale":"2024-07-26T06:13:36.059941+0000","last_undegraded":"2024-07-26T06:13:36.059941+0000","last_fullsized":"2024-07-26T06:13:36.059941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060485+0000","last_change":"2024-07-26T06:13:34.052811+0000","last_active":"2024-07-26T06:13:36.060485+0000","last_peered":"2024-07-26T06:13:36.060485+0000","last_clean":"2024-07-26T06:13:36.060485+0000","last_became_active":"2024-07-26T06:13:34.051989+0000","last_became_peered":"2024-07-26T06:13:34.051989+0000","last_unstale":"2024-07-26T06:13:36.060485+0000","last_undegraded":"2024-07-26T06:13:36.060485+0000","last_fullsized":"2024-07-26T06:13:36.060485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060313+0000","last_change":"2024-07-26T06:13:35.061621+0000","last_active":"2024-07-26T06:13:36.060313+0000","last_peered":"2024-07-26T06:13:36.060313+0000","last_clean":"2024-07-26T06:13:36.060313+0000","last_became_active":"2024-07-26T06:13:35.061349+0000","last_became_peered":"2024-07-26T06:13:35.061349+0000","last_unstale":"2024-07-26T06:13:36.060313+0000","last_undegraded":"2024-07-26T06:13:36.060313+0000","last_fullsized":"2024-07-26T06:13:36.060313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:06:30.885224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059607+0000","last_change":"2024-07-26T06:13:34.069215+0000","last_active":"2024-07-26T06:13:36.059607+0000","last_peered":"2024-07-26T06:13:36.059607+0000","last_clean":"2024-07-26T06:13:36.059607+0000","last_became_active":"2024-07-26T06:13:34.068883+0000","last_became_peered":"2024-07-26T06:13:34.068883+0000","last_unstale":"2024-07-26T06:13:36.059607+0000","last_undegraded":"2024-07-26T06:13:36.059607+0000","last_fullsized":"2024-07-26T06:13:36.059607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:02:49.446347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":657191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":66241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":66367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:13:39.968 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:13:39.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-07-26T06:13:39.968 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-07-26T06:13:39.992 INFO:tasks.ceph.mds.c.smithi163.stderr:2024-07-26T06:13:39.991+0000 7f7e23800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:13:40.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 -- 172.21.15.163:0/3599369099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc0ce4a0 msgr2=0x7f62dc071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 --2- 172.21.15.163:0/3599369099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc0ce4a0 0x7f62dc071530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009830 tx=0x7f62c802a510 comp rx=0 tx=0).stop 2024-07-26T06:13:40.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 -- 172.21.15.163:0/3599369099 shutdown_connections 2024-07-26T06:13:40.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 --2- 172.21.15.163:0/3599369099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc071a70 0x7f62dc0b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 --2- 172.21.15.163:0/3599369099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc0ce4a0 0x7f62dc071530 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 -- 172.21.15.163:0/3599369099 >> 172.21.15.163:0/3599369099 conn(0x7f62dc0b0e30 msgr2=0x7f62dc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:40.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 -- 172.21.15.163:0/3599369099 shutdown_connections 2024-07-26T06:13:40.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.200+0000 7f62db200640 1 -- 172.21.15.163:0/3599369099 wait complete. 2024-07-26T06:13:40.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.202+0000 7f62db200640 1 Processor -- start 2024-07-26T06:13:40.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.202+0000 7f62db200640 1 -- start start 2024-07-26T06:13:40.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62db200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 0x7f62dc1f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62db200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0ce4a0 0x7f62dc1547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62db200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc1556f0 0x7f62dc1f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62db200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc0b3fe0 con 0x7f62dc071a70 2024-07-26T06:13:40.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62db200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc0b3e60 con 0x7f62dc1556f0 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62db200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc0b4160 con 0x7f62dc0ce4a0 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 0x7f62dc1f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 0x7f62dc1f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44874/0 (socket says 172.21.15.163:44874) 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 -- 172.21.15.163:0/1034927103 learned_addr learned my addr 172.21.15.163:0/1034927103 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9400640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0ce4a0 0x7f62dc1547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 -- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0ce4a0 msgr2=0x7f62dc1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0ce4a0 0x7f62dc1547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62da800640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc1556f0 0x7f62dc1f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 -- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc1556f0 msgr2=0x7f62dc1f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc1556f0 0x7f62dc1f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9e00640 1 -- 172.21.15.163:0/1034927103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc152280 con 0x7f62dc071a70 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.204+0000 7f62d9400640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0ce4a0 0x7f62dc1547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.205+0000 7f62d9e00640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 0x7f62dc1f3f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c3a0 tx=0x7f62cc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.205+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc011790 con 0x7f62dc071a70 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.205+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62cc017070 con 0x7f62dc071a70 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.205+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc152560 con 0x7f62dc071a70 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.205+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc011d20 con 0x7f62dc071a70 2024-07-26T06:13:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.205+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc152b20 con 0x7f62dc071a70 2024-07-26T06:13:40.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.206+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc0b46f0 con 0x7f62dc071a70 2024-07-26T06:13:40.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.207+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f62cc01c460 con 0x7f62dc071a70 2024-07-26T06:13:40.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.208+0000 7f62d2000640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62bc0762d0 0x7f62bc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.209+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f62cc0976a0 con 0x7f62dc071a70 2024-07-26T06:13:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.209+0000 7f62d9400640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62bc0762d0 0x7f62bc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.210+0000 7f62d9400640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62bc0762d0 0x7f62bc078790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62c8034000 tx=0x7f62c8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:40.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.212+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62cc060f60 con 0x7f62dc071a70 2024-07-26T06:13:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.422+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f62dc0ced60 con 0x7f62dc071a70 2024-07-26T06:13:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.423+0000 7f62d2000640 1 -- 172.21.15.163:0/1034927103 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7f62cc060860 con 0x7f62dc071a70 2024-07-26T06:13:40.424 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.426+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62bc0762d0 msgr2=0x7f62bc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.426+0000 7f62db200640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62bc0762d0 0x7f62bc078790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62c8034000 tx=0x7f62c8004730 comp rx=0 tx=0).stop 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.426+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 msgr2=0x7f62dc1f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.426+0000 7f62db200640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 0x7f62dc1f3f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c3a0 tx=0x7f62cc00c860 comp rx=0 tx=0).stop 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 shutdown_connections 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62bc0762d0 0x7f62bc078790 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc1556f0 0x7f62dc1f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0ce4a0 0x7f62dc1547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 --2- 172.21.15.163:0/1034927103 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62dc071a70 0x7f62dc1f3f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 >> 172.21.15.163:0/1034927103 conn(0x7f62dc0b0e30 msgr2=0x7f62dc0b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 shutdown_connections 2024-07-26T06:13:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.427+0000 7f62db200640 1 -- 172.21.15.163:0/1034927103 wait complete. 2024-07-26T06:13:40.445 INFO:teuthology.orchestra.run.smithi163.stdout:[{"version":27,"timestamp":"2024-07-26T06:13:21.302433+0000","name":"","changes":[{"name":"osd.6/osd_mclock_max_capacity_iops_ssd","new_value":"3932.844298"}]}] 2024-07-26T06:13:40.445 INFO:tasks.ceph_manager:config epoch is 27 2024-07-26T06:13:40.446 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T06:13:40.446 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-07-26T06:13:40.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-07-26T06:13:40.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 -- 172.21.15.163:0/1055554627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75480052e0 msgr2=0x7f754800fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 --2- 172.21.15.163:0/1055554627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75480052e0 0x7f754800fdd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f755c005fd0 tx=0x7f7548005830 comp rx=0 tx=0).stop 2024-07-26T06:13:40.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 -- 172.21.15.163:0/1055554627 shutdown_connections 2024-07-26T06:13:40.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 --2- 172.21.15.163:0/1055554627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75480052e0 0x7f754800fdd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 -- 172.21.15.163:0/1055554627 >> 172.21.15.163:0/1055554627 conn(0x7f756406d420 msgr2=0x7f756406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:40.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 -- 172.21.15.163:0/1055554627 shutdown_connections 2024-07-26T06:13:40.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.651+0000 7f7569400640 1 -- 172.21.15.163:0/1055554627 wait complete. 2024-07-26T06:13:40.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.653+0000 7f7569400640 1 Processor -- start 2024-07-26T06:13:40.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.653+0000 7f7569400640 1 -- start start 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.654+0000 7f7569400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75480052e0 0x7f75641f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7569400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 0x7f75641554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7569400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75641ef880 0x7f75641efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7569400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f756406c0c0 con 0x7f7564155040 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7569400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f756406bf40 con 0x7f75480052e0 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7569400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f756406c240 con 0x7f75641ef880 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 0x7f75641554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 0x7f75641554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44898/0 (socket says 172.21.15.163:44898) 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 -- 172.21.15.163:0/442218275 learned_addr learned my addr 172.21.15.163:0/442218275 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7562800640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75480052e0 0x7f75641f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7563200640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75641ef880 0x7f75641efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 -- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75641ef880 msgr2=0x7f75641efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75641ef880 0x7f75641efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 -- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75480052e0 msgr2=0x7f75641f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75480052e0 0x7f75641f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f755be00640 1 -- 172.21.15.163:0/442218275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75641f0240 con 0x7f7564155040 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.655+0000 7f7563200640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75641ef880 0x7f75641efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.656+0000 7f755be00640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 0x7f75641554c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f754c00d7d0 tx=0x7f754c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.656+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c018070 con 0x7f7564155040 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.656+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f754c00de80 con 0x7f7564155040 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.656+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c00b6c0 con 0x7f7564155040 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.656+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7564152e40 con 0x7f7564155040 2024-07-26T06:13:40.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.656+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75641f07b0 con 0x7f7564155040 2024-07-26T06:13:40.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.657+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f756415be90 con 0x7f7564155040 2024-07-26T06:13:40.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.658+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f754c01b070 con 0x7f7564155040 2024-07-26T06:13:40.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.659+0000 7f755aa00640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f75400762d0 0x7f7540078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:40.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.659+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f754c096ee0 con 0x7f7564155040 2024-07-26T06:13:40.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.660+0000 7f7562800640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f75400762d0 0x7f7540078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:40.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.661+0000 7f7562800640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f75400762d0 0x7f7540078790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7564152b00 tx=0x7f7554009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:40.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.663+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f754c060150 con 0x7f7564155040 2024-07-26T06:13:40.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.918+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f7564159f70 con 0x7f7564155040 2024-07-26T06:13:40.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.925+0000 7f755aa00640 1 -- 172.21.15.163:0/442218275 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+189526 (secure 0 0 0) 0x7f754c064450 con 0x7f7564155040 2024-07-26T06:13:40.930 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.930+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f75400762d0 msgr2=0x7f7540078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.930+0000 7f7569400640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f75400762d0 0x7f7540078790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7564152b00 tx=0x7f7554009040 comp rx=0 tx=0).stop 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.930+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 msgr2=0x7f75641554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.930+0000 7f7569400640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 0x7f75641554c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f754c00d7d0 tx=0x7f754c00dc90 comp rx=0 tx=0).stop 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 shutdown_connections 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f75400762d0 0x7f7540078790 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75641ef880 0x7f75641efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7564155040 0x7f75641554c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 --2- 172.21.15.163:0/442218275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75480052e0 0x7f75641f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 >> 172.21.15.163:0/442218275 conn(0x7f756406d420 msgr2=0x7f756415a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 shutdown_connections 2024-07-26T06:13:40.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:40.931+0000 7f7569400640 1 -- 172.21.15.163:0/442218275 wait complete. 2024-07-26T06:13:40.950 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":5,"flags":0,"active_gid":4115,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1554374994},{"type":"v1","addr":"172.21.15.164:6817","nonce":1554374994}]},"active_addr":"172.21.15.164:6817/1554374994","active_change":"2024-07-26T06:13:13.625578+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4118,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":83055060}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2203391701}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2212531749}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3425233229}]}]} 2024-07-26T06:13:40.954 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-07-26T06:13:40.954 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-07-26T06:13:40.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:41.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 -- 172.21.15.163:0/2813679016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc88004800 msgr2=0x7fcc88004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 --2- 172.21.15.163:0/2813679016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc88004800 0x7fcc88004c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00c310 tx=0x7fcc88005830 comp rx=0 tx=0).stop 2024-07-26T06:13:41.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 -- 172.21.15.163:0/2813679016 shutdown_connections 2024-07-26T06:13:41.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 --2- 172.21.15.163:0/2813679016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc88004800 0x7fcc88004c80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 -- 172.21.15.163:0/2813679016 >> 172.21.15.163:0/2813679016 conn(0x7fcca40cfda0 msgr2=0x7fcca40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:41.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 -- 172.21.15.163:0/2813679016 shutdown_connections 2024-07-26T06:13:41.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.149+0000 7fcca8a00640 1 -- 172.21.15.163:0/2813679016 wait complete. 2024-07-26T06:13:41.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.151+0000 7fcca8a00640 1 Processor -- start 2024-07-26T06:13:41.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.151+0000 7fcca8a00640 1 -- start start 2024-07-26T06:13:41.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcca8a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc88004800 0x7fcca41f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcca8a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 0x7fcca41544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcca8a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcca4156620 0x7fcca4156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcca8a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcca40ce680 con 0x7fcca41550a0 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcca8a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcca40ce500 con 0x7fcca4156620 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcca8a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcca40ce800 con 0x7fcc88004800 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcc99e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 0x7fcca41544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcc99e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 0x7fcca41544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44904/0 (socket says 172.21.15.163:44904) 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.152+0000 7fcc99e00640 1 -- 172.21.15.163:0/1847199627 learned_addr learned my addr 172.21.15.163:0/1847199627 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcca2800640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc88004800 0x7fcca41f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcca3200640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcca4156620 0x7fcca4156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc99e00640 1 -- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc88004800 msgr2=0x7fcca41f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc99e00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc88004800 0x7fcca41f4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc99e00640 1 -- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcca4156620 msgr2=0x7fcca4156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc99e00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcca4156620 0x7fcca4156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc99e00640 1 -- 172.21.15.163:0/1847199627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca41f4c40 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc99e00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 0x7fcca41544a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c000c00 tx=0x7fcc8c0060d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c031040 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc8c0035f0 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c003a40 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.153+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca41555a0 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.154+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca4155b60 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.155+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca40b6740 con 0x7fcca41550a0 2024-07-26T06:13:41.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.156+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcc8c034070 con 0x7fcca41550a0 2024-07-26T06:13:41.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.157+0000 7fcc9b400640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcc80076300 0x7fcc800787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.157+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc8c0b76e0 con 0x7fcca41550a0 2024-07-26T06:13:41.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.157+0000 7fcca2800640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcc80076300 0x7fcc800787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:41.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.158+0000 7fcca2800640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcc80076300 0x7fcc800787c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcc94006fd0 tx=0x7fcc94006d60 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:41.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.161+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc8c006240 con 0x7fcca41550a0 2024-07-26T06:13:41.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.367+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcca40d03d0 con 0x7fcca41550a0 2024-07-26T06:13:41.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.370+0000 7fcc9b400640 1 -- 172.21.15.163:0/1847199627 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fcc8c081020 con 0x7fcca41550a0 2024-07-26T06:13:41.374 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:41.374 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":21,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:36.040341+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:41.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcc80076300 msgr2=0x7fcc800787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcc80076300 0x7fcc800787c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcc94006fd0 tx=0x7fcc94006d60 comp rx=0 tx=0).stop 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 msgr2=0x7fcca41544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 0x7fcca41544a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c000c00 tx=0x7fcc8c0060d0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 shutdown_connections 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcc80076300 0x7fcc800787c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcca4156620 0x7fcca4156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcca41550a0 0x7fcca41544a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 --2- 172.21.15.163:0/1847199627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc88004800 0x7fcca41f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 >> 172.21.15.163:0/1847199627 conn(0x7fcca40cfda0 msgr2=0x7fcca40b7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 shutdown_connections 2024-07-26T06:13:41.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.373+0000 7fcca8a00640 1 -- 172.21.15.163:0/1847199627 wait complete. 2024-07-26T06:13:41.395 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-07-26T06:13:41.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:41.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.594+0000 7efd03200640 1 -- 172.21.15.163:0/2835562244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 msgr2=0x7efd040b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.594+0000 7efd03200640 1 --2- 172.21.15.163:0/2835562244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd040b64c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efcec005fd0 tx=0x7efcec02a4b0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.595+0000 7efd03200640 1 -- 172.21.15.163:0/2835562244 shutdown_connections 2024-07-26T06:13:41.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.595+0000 7efd03200640 1 --2- 172.21.15.163:0/2835562244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd040b23d0 0x7efd040b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.595+0000 7efd03200640 1 --2- 172.21.15.163:0/2835562244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd040b64c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.595+0000 7efd03200640 1 -- 172.21.15.163:0/2835562244 >> 172.21.15.163:0/2835562244 conn(0x7efd040cfda0 msgr2=0x7efd040d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:41.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.595+0000 7efd03200640 1 -- 172.21.15.163:0/2835562244 shutdown_connections 2024-07-26T06:13:41.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.595+0000 7efd03200640 1 -- 172.21.15.163:0/2835562244 wait complete. 2024-07-26T06:13:41.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.596+0000 7efd03200640 1 Processor -- start 2024-07-26T06:13:41.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.596+0000 7efd03200640 1 -- start start 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.597+0000 7efd03200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd0414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.597+0000 7efd03200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd040b23d0 0x7efd041f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.597+0000 7efd03200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0415d5a0 0x7efd041f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.597+0000 7efd03200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd040b4f20 con 0x7efd040b1e80 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.597+0000 7efd03200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efd040b4da0 con 0x7efd040b23d0 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.597+0000 7efd03200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd040b50a0 con 0x7efd0415d5a0 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd0414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:41.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd0414f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44912/0 (socket says 172.21.15.163:44912) 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 -- 172.21.15.163:0/3634517651 learned_addr learned my addr 172.21.15.163:0/3634517651 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 -- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0415d5a0 msgr2=0x7efd041f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0415d5a0 0x7efd041f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 -- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd040b23d0 msgr2=0x7efd041f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd040b23d0 0x7efd041f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 -- 172.21.15.163:0/3634517651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0415a0d0 con 0x7efd040b1e80 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efd01e00640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd0414f550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c7a0 tx=0x7efcf400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.598+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4017070 con 0x7efd040b1e80 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.599+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcf4005070 con 0x7efd040b1e80 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.599+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0415a3b0 con 0x7efd040b1e80 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.599+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf40054a0 con 0x7efd040b1e80 2024-07-26T06:13:41.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.599+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0414fa90 con 0x7efd040b1e80 2024-07-26T06:13:41.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.600+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd040b5590 con 0x7efd040b1e80 2024-07-26T06:13:41.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.601+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efcf401a070 con 0x7efd040b1e80 2024-07-26T06:13:41.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.602+0000 7efcfaa00640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce00762d0 0x7efce0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:41.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.602+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efcf4096e40 con 0x7efd040b1e80 2024-07-26T06:13:41.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.602+0000 7efcf9400640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce00762d0 0x7efce0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:41.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.603+0000 7efcf9400640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce00762d0 0x7efce0078790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcec033000 tx=0x7efcec0046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:41.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.606+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd040b6f40 con 0x7efd040b1e80 2024-07-26T06:13:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.797+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd040d1a60 con 0x7efd040b1e80 2024-07-26T06:13:41.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.800+0000 7efcfaa00640 1 -- 172.21.15.163:0/3634517651 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7efcf4060790 con 0x7efd040b1e80 2024-07-26T06:13:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:41.801 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":21,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:36.040341+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:41.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.802+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce00762d0 msgr2=0x7efce0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.802+0000 7efd03200640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce00762d0 0x7efce0078790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcec033000 tx=0x7efcec0046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.802+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 msgr2=0x7efd0414f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.802+0000 7efd03200640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd0414f550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c7a0 tx=0x7efcf400cc60 comp rx=0 tx=0).stop 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.802+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 shutdown_connections 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce00762d0 0x7efce0078790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0415d5a0 0x7efd041f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd040b23d0 0x7efd041f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 --2- 172.21.15.163:0/3634517651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd040b1e80 0x7efd0414f550 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 >> 172.21.15.163:0/3634517651 conn(0x7efd040cfda0 msgr2=0x7efd0415c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 shutdown_connections 2024-07-26T06:13:41.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:41.803+0000 7efd03200640 1 -- 172.21.15.163:0/3634517651 wait complete. 2024-07-26T06:13:41.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-07-26T06:13:41.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-07-26T06:13:41.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-07-26T06:13:41.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-07-26T06:13:41.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-07-26T06:13:41.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-07-26T06:13:41.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-07-26T06:13:41.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.118+0000 7fa806a00640 1 -- 172.21.15.163:0/2676426732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f800a100 msgr2=0x7fa7f804eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.118+0000 7fa806a00640 1 --2- 172.21.15.163:0/2676426732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f800a100 0x7fa7f804eb90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa8000511a0 tx=0x7fa8000a5110 comp rx=0 tx=0).stop 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.119+0000 7fa806a00640 1 -- 172.21.15.163:0/2676426732 shutdown_connections 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.119+0000 7fa806a00640 1 --2- 172.21.15.163:0/2676426732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f800a100 0x7fa7f804eb90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.119+0000 7fa806a00640 1 --2- 172.21.15.163:0/2676426732 >> v2:172.21.15.164:3300/0 conn(0x7fa7f8033b10 0x7fa7f8009bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.119+0000 7fa806a00640 1 -- 172.21.15.163:0/2676426732 >> 172.21.15.163:0/2676426732 conn(0x7fa7f801d960 msgr2=0x7fa7f80545d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.119+0000 7fa806a00640 1 -- 172.21.15.163:0/2676426732 shutdown_connections 2024-07-26T06:13:42.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.119+0000 7fa806a00640 1 -- 172.21.15.163:0/2676426732 wait complete. 2024-07-26T06:13:42.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.120+0000 7f00f9600640 1 -- 172.21.15.163:0/1718749196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b23d0 msgr2=0x7f00f40b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.120+0000 7f00f9600640 1 --2- 172.21.15.163:0/1718749196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b23d0 0x7f00f40b6bd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009b60 tx=0x7f00ec02a740 comp rx=0 tx=0).stop 2024-07-26T06:13:42.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.121+0000 7f00f9600640 1 -- 172.21.15.163:0/1718749196 shutdown_connections 2024-07-26T06:13:42.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.121+0000 7f00f9600640 1 --2- 172.21.15.163:0/1718749196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b23d0 0x7f00f40b6bd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.121+0000 7f00f9600640 1 -- 172.21.15.163:0/1718749196 >> 172.21.15.163:0/1718749196 conn(0x7f00f40cfda0 msgr2=0x7f00f40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.121+0000 7f00f9600640 1 -- 172.21.15.163:0/1718749196 shutdown_connections 2024-07-26T06:13:42.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.122+0000 7fa806a00640 1 Processor -- start 2024-07-26T06:13:42.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.122+0000 7fa806a00640 1 -- start start 2024-07-26T06:13:42.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.122+0000 7f00f9600640 1 -- 172.21.15.163:0/1718749196 wait complete. 2024-07-26T06:13:42.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.123+0000 7f36e1000640 1 -- 172.21.15.163:0/789572445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36c0004d40 msgr2=0x7f36c00051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.123+0000 7f36e1000640 1 --2- 172.21.15.163:0/789572445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36c0004d40 0x7f36c00051a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36d400c6f0 tx=0x7f36c0005830 comp rx=0 tx=0).stop 2024-07-26T06:13:42.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.123+0000 7f36dbe00640 1 -- 172.21.15.163:0/789572445 reap_dead start 2024-07-26T06:13:42.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.123+0000 7f36e1000640 1 -- 172.21.15.163:0/789572445 shutdown_connections 2024-07-26T06:13:42.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.123+0000 7f36e1000640 1 -- 172.21.15.163:0/789572445 >> 172.21.15.163:0/789572445 conn(0x7f36dc06bf40 msgr2=0x7f36dc06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.123+0000 7f36e1000640 1 -- 172.21.15.163:0/789572445 shutdown_connections 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7f36e1000640 1 -- 172.21.15.163:0/789572445 wait complete. 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa806a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7f800a100 0x7fa7f80dea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa806a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7f8033b10 0x7fa7f801e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa806a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f8036050 0x7fa7f80e1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa806a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f8033740 con 0x7fa7f800a100 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa806a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f80335c0 con 0x7fa7f8033b10 2024-07-26T06:13:42.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa806a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7f80338c0 con 0x7fa7f8036050 2024-07-26T06:13:42.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7f00f9600640 1 Processor -- start 2024-07-26T06:13:42.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7f00f9600640 1 -- start start 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.124+0000 7fa7ff400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7f8033b10 0x7fa7f801e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.125+0000 7fa7ff400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7f8033b10 0x7fa7f801e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40566/0 (socket says 172.21.15.163:40566) 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.125+0000 7fa7ff400640 1 -- 172.21.15.163:0/2474355675 learned_addr learned my addr 172.21.15.163:0/2474355675 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.125+0000 7fa7ffe00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7f800a100 0x7fa7f80dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.125+0000 7fa804a00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f8036050 0x7fa7f80e1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.125+0000 7fa7ff400640 1 -- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f8036050 msgr2=0x7fa7f80e1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.125+0000 7fa7ff400640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7f8036050 0x7fa7f80e1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7fa7ff400640 1 -- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7f800a100 msgr2=0x7fa7f80dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7fa7ff400640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7f800a100 0x7fa7f80dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7fa7ff400640 1 -- 172.21.15.163:0/2474355675 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f80def50 con 0x7fa7f8033b10 2024-07-26T06:13:42.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f9600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b1e80 0x7f00f40c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7fa7ffe00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7f800a100 0x7fa7f80dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:42.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f9600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00f40b23d0 0x7f00f4157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f9600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00f415d5a0 0x7f00f4157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f9600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f00f40b5130 con 0x7f00f40b23d0 2024-07-26T06:13:42.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f9600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f00f40b4fb0 con 0x7f00f415d5a0 2024-07-26T06:13:42.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f9600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f00f40b52b0 con 0x7f00f40b1e80 2024-07-26T06:13:42.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7fa7ff400640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7f8033b10 0x7fa7f801e9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00e9f0 tx=0x7fa7ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f36e1000640 1 Processor -- start 2024-07-26T06:13:42.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b1e80 0x7f00f40c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.126+0000 7f00f3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00f40b23d0 0x7f00f4157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7f00f3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b1e80 0x7f00f40c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50636/0 (socket says 172.21.15.163:50636) 2024-07-26T06:13:42.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7f00f3e00640 1 -- 172.21.15.163:0/3135285532 learned_addr learned my addr 172.21.15.163:0/3135285532 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec00cd80 con 0x7fa7f8033b10 2024-07-26T06:13:42.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7ec010430 con 0x7fa7f8033b10 2024-07-26T06:13:42.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec010840 con 0x7fa7f8033b10 2024-07-26T06:13:42.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f81870e0 con 0x7fa7f8033b10 2024-07-26T06:13:42.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f81868e0 con 0x7fa7f8033b10 2024-07-26T06:13:42.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7f00f3400640 1 -- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b1e80 msgr2=0x7f00f40c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.128+0000 7f00f3400640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b1e80 0x7f00f40c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.128+0000 7f00f3400640 1 -- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00f415d5a0 msgr2=0x7f00f4157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7f00f8c00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00f415d5a0 0x7f00f4157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.127+0000 7f36e1000640 1 -- start start 2024-07-26T06:13:42.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f36e1000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36c0004d40 0x7f36dc147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f36e1000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36dc14f1d0 0x7f36dc14f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa7f804f300 con 0x7fa7f8033b10 2024-07-26T06:13:42.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f36e1000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36dc14d010 0x7f36dc14d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f36e1000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc15c380 con 0x7f36c0004d40 2024-07-26T06:13:42.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f36e1000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc15c200 con 0x7f36dc14f1d0 2024-07-26T06:13:42.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f36e1000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f36dc15c880 con 0x7f36dc14d010 2024-07-26T06:13:42.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f00f3400640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00f415d5a0 0x7f00f4157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f00f3400640 1 -- 172.21.15.163:0/3135285532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f40c97f0 con 0x7f00f40b23d0 2024-07-26T06:13:42.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f00f3e00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00f40b1e80 0x7f00f40c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.129+0000 7f00f8c00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00f415d5a0 0x7f00f4157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.130+0000 7f36daa00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36dc14f1d0 0x7f36dc14f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.130+0000 7f36dbe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36dc14d010 0x7f36dc14d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.130+0000 7f36daa00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36dc14f1d0 0x7f36dc14f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40576/0 (socket says 172.21.15.163:40576) 2024-07-26T06:13:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.130+0000 7f36daa00640 1 -- 172.21.15.163:0/2505927058 learned_addr learned my addr 172.21.15.163:0/2505927058 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.130+0000 7f00f3400640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00f40b23d0 0x7f00f4157610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00e000c790 tx=0x7f00e000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f36daa00640 1 -- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36dc14d010 msgr2=0x7f36dc14d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f36daa00640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36dc14d010 0x7f36dc14d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f36daa00640 1 -- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36c0004d40 msgr2=0x7f36dc147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f36daa00640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f36c0004d40 0x7f36dc147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f36daa00640 1 -- 172.21.15.163:0/2505927058 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc14fb90 con 0x7f36dc14f1d0 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000a640 con 0x7f00f40b23d0 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00e000a800 con 0x7f00f40b23d0 2024-07-26T06:13:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f4155240 con 0x7f00f40b23d0 2024-07-26T06:13:42.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.131+0000 7f36daa00640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36dc14f1d0 0x7f36dc14f650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011a90 tx=0x7f36c8011f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.132+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000ac90 con 0x7f00f40b23d0 2024-07-26T06:13:42.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.132+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f4155800 con 0x7f00f40b23d0 2024-07-26T06:13:42.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.132+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c800fd90 con 0x7f36dc14f1d0 2024-07-26T06:13:42.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.132+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa7ec0109e0 con 0x7fa7f8033b10 2024-07-26T06:13:42.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.132+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36dc0cae80 con 0x7f36dc14f1d0 2024-07-26T06:13:42.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.132+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36dc0cb440 con 0x7f36dc14f1d0 2024-07-26T06:13:42.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.133+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00f4158090 con 0x7f00f40b23d0 2024-07-26T06:13:42.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.133+0000 7f36d0c00640 1 -- 172.21.15.163:0/2505927058 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f36dc0cbb00 con 0x7f36dc14f1d0 2024-07-26T06:13:42.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.133+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36c8005020 con 0x7f36dc14f1d0 2024-07-26T06:13:42.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.133+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c8005430 con 0x7f36dc14f1d0 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.134+0000 7fa7f3e00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa7e00762d0 0x7fa7e0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.134+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa7ec098ed0 con 0x7fa7f8033b10 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.134+0000 7fa7f3e00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fa7e007e470 0x7fa7e0080890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.134+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 --> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7ec018e50 con 0x7fa7e007e470 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.134+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa7ec0992c0 con 0x7fa7f8033b10 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.135+0000 7fa7ffe00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa7e00762d0 0x7fa7e0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.135+0000 7fa804a00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fa7e007e470 0x7fa7e0080890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.136+0000 7fa804a00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fa7e007e470 0x7fa7e0080890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.137+0000 7fd447200640 1 -- 172.21.15.163:0/438258859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 msgr2=0x7fd4480b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.137+0000 7fd447200640 1 --2- 172.21.15.163:0/438258859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd4480b64c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd434009830 tx=0x7fd43402a400 comp rx=0 tx=0).stop 2024-07-26T06:13:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.137+0000 7fa7ffe00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa7e00762d0 0x7fa7e0078790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4003a70 tx=0x7fa7f4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fd447200640 1 -- 172.21.15.163:0/438258859 shutdown_connections 2024-07-26T06:13:42.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== osd.1 v2:172.21.15.163:6816/2784916277 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7ec018e50 con 0x7fa7e007e470 2024-07-26T06:13:42.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fd447200640 1 --2- 172.21.15.163:0/438258859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4480b23d0 0x7fd4480b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fd447200640 1 --2- 172.21.15.163:0/438258859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd4480b64c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fd447200640 1 -- 172.21.15.163:0/438258859 >> 172.21.15.163:0/438258859 conn(0x7fd4480cfda0 msgr2=0x7fd4480d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fd447200640 1 -- 172.21.15.163:0/438258859 shutdown_connections 2024-07-26T06:13:42.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7fd447200640 1 -- 172.21.15.163:0/438258859 wait complete. 2024-07-26T06:13:42.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.136+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00e001b400 con 0x7f00f40b23d0 2024-07-26T06:13:42.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.137+0000 7f00ebe00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00d40763a0 0x7f00d4078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f00e00976a0 con 0x7f00f40b23d0 2024-07-26T06:13:42.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7f00ebe00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7f00d407e540 0x7f00d4080960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 --> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00d4081010 con 0x7f00d407e540 2024-07-26T06:13:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.138+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f36c80047a0 con 0x7f36dc14f1d0 2024-07-26T06:13:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.139+0000 7f00f8c00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7f00d407e540 0x7f00d4080960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.139+0000 7f00f3e00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00d40763a0 0x7f00d4078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.140+0000 7feff2400640 1 -- 172.21.15.163:0/1498842499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 msgr2=0x7fefec06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.140+0000 7feff2400640 1 --2- 172.21.15.163:0/1498842499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 0x7fefec06bf40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fefd8007350 tx=0x7fefd802b150 comp rx=0 tx=0).stop 2024-07-26T06:13:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.140+0000 7f00f8c00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7f00d407e540 0x7f00d4080960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f00e009e050 con 0x7f00f40b23d0 2024-07-26T06:13:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7feff2400640 1 -- 172.21.15.163:0/1498842499 shutdown_connections 2024-07-26T06:13:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7feff2400640 1 --2- 172.21.15.163:0/1498842499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 0x7fefec06bf40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7feff2400640 1 --2- 172.21.15.163:0/1498842499 >> v2:172.21.15.163:3300/0 conn(0x7fefec159380 0x7fefec0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7feff2400640 1 -- 172.21.15.163:0/1498842499 >> 172.21.15.163:0/1498842499 conn(0x7fefec06d420 msgr2=0x7fefec06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.142+0000 7feff2400640 1 -- 172.21.15.163:0/1498842499 shutdown_connections 2024-07-26T06:13:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.142+0000 7feff2400640 1 -- 172.21.15.163:0/1498842499 wait complete. 2024-07-26T06:13:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7fd447200640 1 Processor -- start 2024-07-26T06:13:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.141+0000 7fd447200640 1 -- start start 2024-07-26T06:13:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.142+0000 7f00f3e00640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00d40763a0 0x7f00d4078860 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f00e4007fd0 tx=0x7f00e4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.142+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== osd.0 v2:172.21.15.163:6800/337444050 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00d4081010 con 0x7f00d407e540 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.142+0000 7fd447200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd448070180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd447200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4480b23d0 0x7fd448155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd447200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815d5a0 0x7fd448203210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd447200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd4480b4f20 con 0x7fd4480b1e80 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd447200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd4480b4da0 con 0x7fd44815d5a0 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd447200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd4480b50a0 con 0x7fd4480b23d0 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd445e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd448070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd445e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd448070180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:44980/0 (socket says 172.21.15.163:44980) 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.143+0000 7fd445e00640 1 -- 172.21.15.163:0/1248753213 learned_addr learned my addr 172.21.15.163:0/1248753213 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd446800640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815d5a0 0x7fd448203210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd445400640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4480b23d0 0x7fd448155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd445e00640 1 -- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4480b23d0 msgr2=0x7fd448155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd445e00640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4480b23d0 0x7fd448155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd445e00640 1 -- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815d5a0 msgr2=0x7fd448203210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd445e00640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815d5a0 0x7fd448203210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.144+0000 7fd445e00640 1 -- 172.21.15.163:0/1248753213 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd448070a80 con 0x7fd4480b1e80 2024-07-26T06:13:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7feff2400640 1 Processor -- start 2024-07-26T06:13:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7fd445e00640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd448070180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c3a0 tx=0x7fd43800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7feff2400640 1 -- start start 2024-07-26T06:13:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438011790 con 0x7fd4480b1e80 2024-07-26T06:13:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd438017070 con 0x7fd4480b1e80 2024-07-26T06:13:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438011cc0 con 0x7fd4480b1e80 2024-07-26T06:13:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd448071730 con 0x7fd4480b1e80 2024-07-26T06:13:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7feff2400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec06c480 0x7fefec14ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7feff2400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 0x7fefec146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7feff2400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefec159380 0x7fefec157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7feff2400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fefec0cd8f0 con 0x7fefec06c480 2024-07-26T06:13:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7feff2400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fefec06f580 con 0x7fefec159380 2024-07-26T06:13:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7feff2400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefec0cda70 con 0x7fefec0b40a0 2024-07-26T06:13:42.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd448071cf0 con 0x7fd4480b1e80 2024-07-26T06:13:42.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 -- 172.21.15.163:0/2223888439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0e4157f70 msgr2=0x7ff0e415ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 --2- 172.21.15.163:0/2223888439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0e4157f70 0x7ff0e415ba40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4074970 tx=0x7ff0d402a400 comp rx=0 tx=0).stop 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7fefeb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec06c480 0x7fefec14ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7fefeb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec06c480 0x7fefec14ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45006/0 (socket says 172.21.15.163:45006) 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7fefeb400640 1 -- 172.21.15.163:0/3957537144 learned_addr learned my addr 172.21.15.163:0/3957537144 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7fefebe00640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefec159380 0x7fefec157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 -- 172.21.15.163:0/2223888439 shutdown_connections 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 --2- 172.21.15.163:0/2223888439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e40cbfc0 0x7ff0e40cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 --2- 172.21.15.163:0/2223888439 >> v2:172.21.15.164:3300/0 conn(0x7ff0e415a880 0x7ff0e415acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 --2- 172.21.15.163:0/2223888439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0e4157f70 0x7ff0e415ba40 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4074970 tx=0x7ff0d402a400 comp rx=0 tx=0).stop 2024-07-26T06:13:42.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7ff0e9400640 1 -- 172.21.15.163:0/2223888439 >> 172.21.15.163:0/2223888439 conn(0x7ff0e40b5950 msgr2=0x7ff0e40b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7fefeaa00640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 0x7fefec146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 -- 172.21.15.163:0/2666654293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b53b0 msgr2=0x7fd0040b57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 --2- 172.21.15.163:0/2666654293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b53b0 0x7fd0040b57d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcffc005fd0 tx=0x7fcffc02ab30 comp rx=0 tx=0).stop 2024-07-26T06:13:42.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd4480b5590 con 0x7fd4480b1e80 2024-07-26T06:13:42.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 -- 172.21.15.163:0/2666654293 shutdown_connections 2024-07-26T06:13:42.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 --2- 172.21.15.163:0/2666654293 >> v2:172.21.15.164:3300/0 conn(0x7fd0040b6720 0x7fd0040b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 --2- 172.21.15.163:0/2666654293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b53b0 0x7fd0040b57d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 -- 172.21.15.163:0/2666654293 >> 172.21.15.163:0/2666654293 conn(0x7fd0040d36b0 msgr2=0x7fd0040d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 -- 172.21.15.163:0/2666654293 shutdown_connections 2024-07-26T06:13:42.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fd00c600640 1 -- 172.21.15.163:0/2666654293 wait complete. 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fefeb400640 1 -- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 msgr2=0x7fefec146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fefeb400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefec0b40a0 0x7fefec146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fefeb400640 1 -- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefec159380 msgr2=0x7fefec157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fefeb400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefec159380 0x7fefec157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7fefeb400640 1 -- 172.21.15.163:0/3957537144 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefec1fba50 con 0x7fefec06c480 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.149+0000 7ff0e9400640 1 -- 172.21.15.163:0/2223888439 shutdown_connections 2024-07-26T06:13:42.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.149+0000 7ff0e9400640 1 -- 172.21.15.163:0/2223888439 wait complete. 2024-07-26T06:13:42.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f654ae00640 1 -- 172.21.15.163:0/2280934784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 msgr2=0x7f654406c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f654ae00640 1 --2- 172.21.15.163:0/2280934784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 0x7f654406c480 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f653c005fd0 tx=0x7f653c02aac0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7fd00c600640 1 Processor -- start 2024-07-26T06:13:42.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f654ae00640 1 -- 172.21.15.163:0/2280934784 shutdown_connections 2024-07-26T06:13:42.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f654ae00640 1 --2- 172.21.15.163:0/2280934784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 0x7f654406c480 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f654ae00640 1 --2- 172.21.15.163:0/2280934784 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6544161a70 0x7f654406bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f654ae00640 1 -- 172.21.15.163:0/2280934784 >> 172.21.15.163:0/2280934784 conn(0x7f654406d420 msgr2=0x7f654406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7fd00c600640 1 -- start start 2024-07-26T06:13:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.151+0000 7ff0e9400640 1 Processor -- start 2024-07-26T06:13:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.151+0000 7ff0e9400640 1 -- start start 2024-07-26T06:13:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.151+0000 7fefeb400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec06c480 0x7fefec14ab90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00c970 tx=0x7fefdc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.151+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd43801c460 con 0x7fd4480b1e80 2024-07-26T06:13:42.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.151+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc007bb0 con 0x7fefec06c480 2024-07-26T06:13:42.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.151+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefec1fbd30 con 0x7fefec06c480 2024-07-26T06:13:42.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefec1fc2f0 con 0x7fefec06c480 2024-07-26T06:13:42.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7ff0e9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0e40cbfc0 0x7ff0e4146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7ff0e9400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e415a880 0x7ff0e41e9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7ff0e9400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e4146bb0 0x7ff0e4147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7ff0e9400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e40b3fe0 con 0x7ff0e40cbfc0 2024-07-26T06:13:42.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7ff0e9400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e40b3e60 con 0x7ff0e4146bb0 2024-07-26T06:13:42.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.152+0000 7ff0e9400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e40b4160 con 0x7ff0e415a880 2024-07-26T06:13:42.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd43e000640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd428076320 0x7fd4280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7ff0e3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e4146bb0 0x7ff0e4147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd438097e50 con 0x7fd4480b1e80 2024-07-26T06:13:42.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd00c600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0040b53b0 0x7fd004151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7ff0e1e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e415a880 0x7ff0e41e9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7ff0e3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e4146bb0 0x7ff0e4147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40616/0 (socket says 172.21.15.163:40616) 2024-07-26T06:13:42.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7ff0e1e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e415a880 0x7ff0e41e9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50700/0 (socket says 172.21.15.163:50700) 2024-07-26T06:13:42.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7ff0e3200640 1 -- 172.21.15.163:0/4172072315 learned_addr learned my addr 172.21.15.163:0/4172072315 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd43e000640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7fd42807e5a0 0x7fd4280809c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 --> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd438025e50 con 0x7fd42807e5a0 2024-07-26T06:13:42.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd438098330 con 0x7fd4480b1e80 2024-07-26T06:13:42.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7fd445400640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd428076320 0x7fd4280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7f36d3400640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f36bc076300 0x7f36bc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.145+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f36c8098a30 con 0x7f36dc14f1d0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7f36d3400640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7f36bc07e520 0x7f36bc080940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 --> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36bc080ff0 con 0x7f36bc07e520 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.146+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f36c8098e20 con 0x7f36dc14f1d0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7f36db400640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f36bc076300 0x7f36bc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.147+0000 7f36dbe00640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7f36bc07e520 0x7f36bc080940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7f36dbe00640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7f36bc07e520 0x7f36bc080940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.148+0000 7f36db400640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f36bc076300 0x7f36bc0787c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0ccba0 tx=0x7f36cc006a40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.150+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== osd.2 v2:172.21.15.163:6808/1222541827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36bc080ff0 con 0x7f36bc07e520 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e3200640 1 -- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e415a880 msgr2=0x7ff0e41e9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e3200640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e415a880 0x7ff0e41e9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e3200640 1 -- 172.21.15.163:0/4172072315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0e40cbfc0 msgr2=0x7ff0e4146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e3200640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0e40cbfc0 0x7ff0e4146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e3200640 1 -- 172.21.15.163:0/4172072315 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e41ea720 con 0x7ff0e4146bb0 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.153+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fefec0503f0 con 0x7fefec06c480 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefdc007d50 con 0x7fefec06c480 2024-07-26T06:13:42.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc005260 con 0x7fefec06c480 2024-07-26T06:13:42.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7fd00c600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b6720 0x7fd00406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7fd446800640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7fd42807e5a0 0x7fd4280809c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e1e00640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e415a880 0x7ff0e41e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-07-26T06:13:42.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7ff0e3200640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e4146bb0 0x7ff0e4147310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00ef40 tx=0x7ff0dc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc004b20 con 0x7ff0e4146bb0 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0dc004dd0 con 0x7ff0e4146bb0 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc003bd0 con 0x7ff0e4146bb0 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e41eaa00 con 0x7ff0e4146bb0 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7fd445400640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd428076320 0x7fd4280787e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd434034000 tx=0x7fd434004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7fd00c600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd00415d650 0x7fd00406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7fd009200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b6720 0x7fd00406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7fd00c600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd0040b7250 con 0x7fd0040b53b0 2024-07-26T06:13:42.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7fd00c600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd0040b70d0 con 0x7fd00415d650 2024-07-26T06:13:42.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7fd00c600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd0040b73d0 con 0x7fd0040b6720 2024-07-26T06:13:42.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.156+0000 7f654ae00640 1 -- 172.21.15.163:0/2280934784 shutdown_connections 2024-07-26T06:13:42.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.155+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e41eafc0 con 0x7ff0e4146bb0 2024-07-26T06:13:42.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.156+0000 7f654ae00640 1 -- 172.21.15.163:0/2280934784 wait complete. 2024-07-26T06:13:42.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.156+0000 7fd446800640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7fd42807e5a0 0x7fd4280809c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.156+0000 7fd009200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b6720 0x7fd00406f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50710/0 (socket says 172.21.15.163:50710) 2024-07-26T06:13:42.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.156+0000 7fd009200640 1 -- 172.21.15.163:0/2529241118 learned_addr learned my addr 172.21.15.163:0/2529241118 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.156+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0e41eb3a0 con 0x7ff0e4146bb0 2024-07-26T06:13:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.154+0000 7fd009c00640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0040b53b0 0x7fd004151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd009200640 1 -- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd00415d650 msgr2=0x7fd00406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd009200640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd00415d650 0x7fd00406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd009200640 1 -- 172.21.15.163:0/2529241118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0040b53b0 msgr2=0x7fd004151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd009200640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0040b53b0 0x7fd004151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd009200640 1 -- 172.21.15.163:0/2529241118 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0040ccf40 con 0x7fd0040b6720 2024-07-26T06:13:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== osd.3 v2:172.21.15.163:6824/136015621 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd438025e50 con 0x7fd42807e5a0 2024-07-26T06:13:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.157+0000 7fd009200640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b6720 0x7fd00406f260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcffc000f80 tx=0x7fcffc02cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.158+0000 7f654ae00640 1 Processor -- start 2024-07-26T06:13:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.158+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc004450 con 0x7fd0040b6720 2024-07-26T06:13:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.158+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcffc043070 con 0x7fd0040b6720 2024-07-26T06:13:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.158+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc02d460 con 0x7fd0040b6720 2024-07-26T06:13:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.158+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0040cc3c0 con 0x7fd0040b6720 2024-07-26T06:13:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.159+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0040cc980 con 0x7fd0040b6720 2024-07-26T06:13:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.160+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd0040ca180 con 0x7fd0040b6720 2024-07-26T06:13:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.160+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fefdc0054c0 con 0x7fefec06c480 2024-07-26T06:13:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.160+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff0dc01d020 con 0x7ff0e4146bb0 2024-07-26T06:13:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.159+0000 7f654ae00640 1 -- start start 2024-07-26T06:13:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.160+0000 7f654ae00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6544073c00 0x7f65440ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.160+0000 7f654ae00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 0x7f65441f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.161+0000 7f654ae00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65441f76b0 0x7f65440a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7f6543400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 0x7f65441f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7f6548e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65441f76b0 0x7f65440a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7f6548e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65441f76b0 0x7f65440a9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40640/0 (socket says 172.21.15.163:40640) 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.163+0000 7f6548e00640 1 -- 172.21.15.163:0/2321303704 learned_addr learned my addr 172.21.15.163:0/2321303704 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.163+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6544071c20 con 0x7f65440ccc60 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.163+0000 7f6543e00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6544073c00 0x7f65440ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.163+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6544071aa0 con 0x7f65441f76b0 2024-07-26T06:13:42.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.163+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6544071da0 con 0x7f6544073c00 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7ff0daa00640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0c40762d0 0x7ff0c4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff0dc0988c0 con 0x7ff0e4146bb0 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7ff0daa00640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff0c407e4f0 0x7ff0c4080910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 --> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0dc017ec0 con 0x7ff0c407e4f0 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.162+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff0dc098cb0 con 0x7ff0e4146bb0 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.163+0000 7ff0e2800640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0c40762d0 0x7ff0c4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.164+0000 7ff0e2800640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0c40762d0 0x7ff0c4078790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4034000 tx=0x7ff0d40046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7fefe3400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fefcc076300 0x7fefcc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7ff0e1e00640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff0c407e4f0 0x7ff0c4080910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.164+0000 7f6543400640 1 -- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6544073c00 msgr2=0x7f65440ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.164+0000 7f6543400640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6544073c00 0x7f65440ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.164+0000 7f6543400640 1 -- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65441f76b0 msgr2=0x7f65440a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.164+0000 7f6543400640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65441f76b0 0x7f65440a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.164+0000 7f6543400640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65440aad30 con 0x7f65440ccc60 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7f6548e00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65441f76b0 0x7f65440a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.166+0000 7f6543400640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 0x7f65441f7170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f653000c390 tx=0x7f653000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fefdc097ef0 con 0x7fefec06c480 2024-07-26T06:13:42.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7fefe3400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7fefcc07e520 0x7fefcc080940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 --> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefdc01fec0 con 0x7fefcc07e520 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fefdc09a550 con 0x7fefec06c480 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.165+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcffc036400 con 0x7fd0040b6720 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.166+0000 7fd001600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcfe4076320 0x7fcfe40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.166+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcffc0b9270 con 0x7fd0040b6720 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7fd001600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7fcfe407e540 0x7fcfe4080960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7fd009c00640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcfe4076320 0x7fcfe40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 --> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfe4081010 con 0x7fcfe407e540 2024-07-26T06:13:42.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcffc0b9620 con 0x7fd0040b6720 2024-07-26T06:13:42.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6530017070 con 0x7f65440ccc60 2024-07-26T06:13:42.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7fd00a600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7fcfe407e540 0x7fcfe4080960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65440ab7a0 con 0x7f65440ccc60 2024-07-26T06:13:42.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65441f7b10 con 0x7f65440ccc60 2024-07-26T06:13:42.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65300053e0 con 0x7f65440ccc60 2024-07-26T06:13:42.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.167+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6530003850 con 0x7f65440ccc60 2024-07-26T06:13:42.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.168+0000 7fd009c00640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcfe4076320 0x7fcfe40787e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcff00015e0 tx=0x7fcff0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.168+0000 7fefebe00640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7fefcc07e520 0x7fefcc080940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.168+0000 7fefeaa00640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fefcc076300 0x7fefcc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.168+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65440abe60 con 0x7f65440ccc60 2024-07-26T06:13:42.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.169+0000 7fd00a600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7fcfe407e540 0x7fcfe4080960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.171+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== osd.6 v2:172.21.15.164:6808/2876752734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfe4081010 con 0x7fcfe407e540 2024-07-26T06:13:42.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.170+0000 7fefebe00640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7fefcc07e520 0x7fefcc080940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.170+0000 7fefeaa00640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fefcc076300 0x7fefcc0787c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fefd8034000 tx=0x7fefd802d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.172+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f653001a070 con 0x7f65440ccc60 2024-07-26T06:13:42.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.173+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== osd.4 v2:172.21.15.164:6826/3692835080 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fefdc01fec0 con 0x7fefcc07e520 2024-07-26T06:13:42.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.174+0000 7f653be00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f65240763a0 0x7f6524078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.174+0000 7ff0e1e00640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff0c407e4f0 0x7ff0c4080910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.174+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f65300971e0 con 0x7f65440ccc60 2024-07-26T06:13:42.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.175+0000 7f653be00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f652407e5c0 0x7f65240809e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.175+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6530013e50 con 0x7f652407e5c0 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.175+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f65300975d0 con 0x7f65440ccc60 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.175+0000 7f6543e00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f65240763a0 0x7f6524078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.175+0000 7f6548e00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f652407e5c0 0x7f65240809e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.176+0000 7f6543e00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f65240763a0 0x7f6524078860 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f654406e2a0 tx=0x7f6534008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.177+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== osd.5 v2:172.21.15.164:6800/3774389112 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0dc017ec0 con 0x7ff0c407e4f0 2024-07-26T06:13:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.177+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 --> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7f81863e0 con 0x7fa7e007e470 2024-07-26T06:13:42.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.177+0000 7f6548e00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f652407e5c0 0x7f65240809e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.178+0000 7fa7f3e00640 1 -- 172.21.15.163:0/2474355675 <== osd.1 v2:172.21.15.163:6816/2784916277 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa7f81863e0 con 0x7fa7e007e470 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.178+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fa7e007e470 msgr2=0x7fa7e0080890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.178+0000 7fa806a00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fa7e007e470 0x7fa7e0080890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa7e00762d0 msgr2=0x7fa7e0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa7e00762d0 0x7fa7e0078790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4003a70 tx=0x7fa7f4009040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7f8033b10 msgr2=0x7fa7f801e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 --2- 172.21.15.163:0/2474355675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7f8033b10 0x7fa7f801e9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00e9f0 tx=0x7fa7ec00eeb0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa804a00640 1 -- 172.21.15.163:0/2474355675 reap_dead start 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 shutdown_connections 2024-07-26T06:13:42.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 >> 172.21.15.163:0/2474355675 conn(0x7fa7f801d960 msgr2=0x7fa7f804adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 shutdown_connections 2024-07-26T06:13:42.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.179+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 --> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f00f40b5830 con 0x7f00d407e540 2024-07-26T06:13:42.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.180+0000 7f00ebe00640 1 -- 172.21.15.163:0/3135285532 <== osd.0 v2:172.21.15.163:6800/337444050 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f00f40b5830 con 0x7f00d407e540 2024-07-26T06:13:42.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.180+0000 7fa806a00640 1 -- 172.21.15.163:0/2474355675 wait complete. 2024-07-26T06:13:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.180+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== osd.7 v2:172.21.15.164:6818/734207166 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6530013e50 con 0x7f652407e5c0 2024-07-26T06:13:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.180+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7f00d407e540 msgr2=0x7f00d4080960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.180+0000 7f00f9600640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7f00d407e540 0x7f00d4080960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00d40763a0 msgr2=0x7f00d4078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f9600640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00d40763a0 0x7f00d4078860 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f00e4007fd0 tx=0x7f00e4009040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00f40b23d0 msgr2=0x7f00f4157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f9600640 1 --2- 172.21.15.163:0/3135285532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00f40b23d0 0x7f00f4157610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00e000c790 tx=0x7f00e000cc50 comp rx=0 tx=0).stop 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f8c00640 1 -- 172.21.15.163:0/3135285532 reap_dead start 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 shutdown_connections 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.181+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 >> 172.21.15.163:0/3135285532 conn(0x7f00f40cfda0 msgr2=0x7f00f40b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.182+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 shutdown_connections 2024-07-26T06:13:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.182+0000 7f00f9600640 1 -- 172.21.15.163:0/3135285532 wait complete. 2024-07-26T06:13:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.190+0000 7f36d0c00640 1 -- 172.21.15.163:0/2505927058 --> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f36dc0d1630 con 0x7f36bc07e520 2024-07-26T06:13:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.190+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 --> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd44815b1d0 con 0x7fd42807e5a0 2024-07-26T06:13:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.191+0000 7f36d3400640 1 -- 172.21.15.163:0/2505927058 <== osd.2 v2:172.21.15.163:6808/1222541827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f36dc0d1630 con 0x7f36bc07e520 2024-07-26T06:13:42.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.192+0000 7fd43e000640 1 -- 172.21.15.163:0/1248753213 <== osd.3 v2:172.21.15.163:6824/136015621 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd44815b1d0 con 0x7fd42807e5a0 2024-07-26T06:13:42.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.192+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 --> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fefec06bf40 con 0x7fefcc07e520 2024-07-26T06:13:42.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.192+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7f36bc07e520 msgr2=0x7f36bc080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.192+0000 7f36e1000640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7f36bc07e520 0x7f36bc080940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.192+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7fd42807e5a0 msgr2=0x7fd4280809c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f36bc076300 msgr2=0x7f36bc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36e1000640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f36bc076300 0x7f36bc0787c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0ccba0 tx=0x7f36cc006a40 comp rx=0 tx=0).stop 2024-07-26T06:13:42.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.192+0000 7fd447200640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7fd42807e5a0 0x7fd4280809c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36dc14f1d0 msgr2=0x7f36dc14f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36e1000640 1 --2- 172.21.15.163:0/2505927058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36dc14f1d0 0x7f36dc14f650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f36c8011a90 tx=0x7f36c8011f50 comp rx=0 tx=0).stop 2024-07-26T06:13:42.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd428076320 msgr2=0x7fd4280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7fd447200640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd428076320 0x7fd4280787e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd434034000 tx=0x7fd434004730 comp rx=0 tx=0).stop 2024-07-26T06:13:42.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 msgr2=0x7fd448070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7fd447200640 1 --2- 172.21.15.163:0/1248753213 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4480b1e80 0x7fd448070180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c3a0 tx=0x7fd43800c860 comp rx=0 tx=0).stop 2024-07-26T06:13:42.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36dbe00640 1 -- 172.21.15.163:0/2505927058 reap_dead start 2024-07-26T06:13:42.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 shutdown_connections 2024-07-26T06:13:42.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 >> 172.21.15.163:0/2505927058 conn(0x7f36dc06bf40 msgr2=0x7f36dc06dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7fd446800640 1 -- 172.21.15.163:0/1248753213 reap_dead start 2024-07-26T06:13:42.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.193+0000 7fefe3400640 1 -- 172.21.15.163:0/3957537144 <== osd.4 v2:172.21.15.164:6826/3692835080 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fefec06bf40 con 0x7fefcc07e520 2024-07-26T06:13:42.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 shutdown_connections 2024-07-26T06:13:42.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 >> 172.21.15.163:0/1248753213 conn(0x7fd4480cfda0 msgr2=0x7fd44815a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 shutdown_connections 2024-07-26T06:13:42.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7fd447200640 1 -- 172.21.15.163:0/1248753213 wait complete. 2024-07-26T06:13:42.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7fefcc07e520 msgr2=0x7fefcc080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7feff2400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7fefcc07e520 0x7fefcc080940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fefcc076300 msgr2=0x7fefcc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7feff2400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fefcc076300 0x7fefcc0787c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fefd8034000 tx=0x7fefd802d040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec06c480 msgr2=0x7fefec14ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.194+0000 7feff2400640 1 --2- 172.21.15.163:0/3957537144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec06c480 0x7fefec14ab90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00c970 tx=0x7fefdc00ce30 comp rx=0 tx=0).stop 2024-07-26T06:13:42.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 shutdown_connections 2024-07-26T06:13:42.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7f36e1000640 1 -- 172.21.15.163:0/2505927058 wait complete. 2024-07-26T06:13:42.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7fefebe00640 1 -- 172.21.15.163:0/3957537144 reap_dead start 2024-07-26T06:13:42.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 shutdown_connections 2024-07-26T06:13:42.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 >> 172.21.15.163:0/3957537144 conn(0x7fefec06d420 msgr2=0x7fefec156970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 shutdown_connections 2024-07-26T06:13:42.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.195+0000 7feff2400640 1 -- 172.21.15.163:0/3957537144 wait complete. 2024-07-26T06:13:42.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.200+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 --> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd00414f8b0 con 0x7fcfe407e540 2024-07-26T06:13:42.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.203+0000 7fd001600640 1 -- 172.21.15.163:0/2529241118 <== osd.6 v2:172.21.15.164:6808/2876752734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd00414f8b0 con 0x7fcfe407e540 2024-07-26T06:13:42.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.203+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7fcfe407e540 msgr2=0x7fcfe4080960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.203+0000 7fd00c600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7fcfe407e540 0x7fcfe4080960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.208+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcfe4076320 msgr2=0x7fcfe40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.208+0000 7fd00c600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcfe4076320 0x7fcfe40787e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcff00015e0 tx=0x7fcff0009040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.208+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b6720 msgr2=0x7fd00406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.208+0000 7fd00c600640 1 --2- 172.21.15.163:0/2529241118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0040b6720 0x7fd00406f260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcffc000f80 tx=0x7fcffc02cf00 comp rx=0 tx=0).stop 2024-07-26T06:13:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.208+0000 7fd00a600640 1 -- 172.21.15.163:0/2529241118 reap_dead start 2024-07-26T06:13:42.211 INFO:teuthology.orchestra.run.smithi163.stdout:60129542152 2024-07-26T06:13:42.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-07-26T06:13:42.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.210+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 shutdown_connections 2024-07-26T06:13:42.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.210+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 >> 172.21.15.163:0/2529241118 conn(0x7fd0040d36b0 msgr2=0x7fd0040ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.210+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 shutdown_connections 2024-07-26T06:13:42.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.210+0000 7fd00c600640 1 -- 172.21.15.163:0/2529241118 wait complete. 2024-07-26T06:13:42.213 INFO:teuthology.orchestra.run.smithi163.stdout:64424509447 2024-07-26T06:13:42.213 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.213+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 --> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff0e40cc410 con 0x7ff0c407e4f0 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.214+0000 7ff0daa00640 1 -- 172.21.15.163:0/4172072315 <== osd.5 v2:172.21.15.164:6800/3774389112 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff0e40cc410 con 0x7ff0c407e4f0 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff0c407e4f0 msgr2=0x7ff0c4080910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff0c407e4f0 0x7ff0c4080910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0c40762d0 msgr2=0x7ff0c4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0c40762d0 0x7ff0c4078790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4034000 tx=0x7ff0d40046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e4146bb0 msgr2=0x7ff0e4147310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 --2- 172.21.15.163:0/4172072315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e4146bb0 0x7ff0e4147310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00ef40 tx=0x7ff0dc00c800 comp rx=0 tx=0).stop 2024-07-26T06:13:42.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e3200640 1 -- 172.21.15.163:0/4172072315 reap_dead start 2024-07-26T06:13:42.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 shutdown_connections 2024-07-26T06:13:42.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 >> 172.21.15.163:0/4172072315 conn(0x7ff0e40b5950 msgr2=0x7ff0e40b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 shutdown_connections 2024-07-26T06:13:42.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.215+0000 7ff0e9400640 1 -- 172.21.15.163:0/4172072315 wait complete. 2024-07-26T06:13:42.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.214+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 --> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6544072290 con 0x7f652407e5c0 2024-07-26T06:13:42.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.216+0000 7f653be00640 1 -- 172.21.15.163:0/2321303704 <== osd.7 v2:172.21.15.164:6818/734207166 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6544072290 con 0x7f652407e5c0 2024-07-26T06:13:42.220 INFO:teuthology.orchestra.run.smithi163.stdout:60129542151 2024-07-26T06:13:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f652407e5c0 msgr2=0x7f65240809e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f652407e5c0 0x7f65240809e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f65240763a0 msgr2=0x7f6524078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f65240763a0 0x7f6524078860 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f654406e2a0 tx=0x7f6534008040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 msgr2=0x7f65441f7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 --2- 172.21.15.163:0/2321303704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65440ccc60 0x7f65441f7170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f653000c390 tx=0x7f653000c850 comp rx=0 tx=0).stop 2024-07-26T06:13:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f6548e00640 1 -- 172.21.15.163:0/2321303704 reap_dead start 2024-07-26T06:13:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 shutdown_connections 2024-07-26T06:13:42.227 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-07-26T06:13:42.227 INFO:teuthology.orchestra.run.smithi163.stdout:64424509448 2024-07-26T06:13:42.228 INFO:teuthology.orchestra.run.smithi163.stdout:60129542150 2024-07-26T06:13:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.220+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 >> 172.21.15.163:0/2321303704 conn(0x7f654406d420 msgr2=0x7f6544161f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.221+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 shutdown_connections 2024-07-26T06:13:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.221+0000 7f654ae00640 1 -- 172.21.15.163:0/2321303704 wait complete. 2024-07-26T06:13:42.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-07-26T06:13:42.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-07-26T06:13:42.235 INFO:teuthology.orchestra.run.smithi163.stdout:64424509447 2024-07-26T06:13:42.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-07-26T06:13:42.236 INFO:teuthology.orchestra.run.smithi163.stdout:64424509446 2024-07-26T06:13:42.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-07-26T06:13:42.246 INFO:teuthology.orchestra.run.smithi163.stdout:64424509446 2024-07-26T06:13:42.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-07-26T06:13:42.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.479+0000 7fced0a00640 1 -- 172.21.15.163:0/2107973229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec40e0a90 msgr2=0x7fcec4053ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.479+0000 7fced0a00640 1 --2- 172.21.15.163:0/2107973229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec40e0a90 0x7fcec4053ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcec00098e0 tx=0x7fcec002a440 comp rx=0 tx=0).stop 2024-07-26T06:13:42.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.479+0000 7fced0a00640 1 -- 172.21.15.163:0/2107973229 shutdown_connections 2024-07-26T06:13:42.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.480+0000 7fced0a00640 1 --2- 172.21.15.163:0/2107973229 >> v2:172.21.15.164:3300/0 conn(0x7fcec4039960 0x7fcec404eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.480+0000 7fced0a00640 1 --2- 172.21.15.163:0/2107973229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec40e0a90 0x7fcec4053ce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.480+0000 7fced0a00640 1 -- 172.21.15.163:0/2107973229 >> 172.21.15.163:0/2107973229 conn(0x7fcec401d930 msgr2=0x7fcec40545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.481+0000 7fced0a00640 1 -- 172.21.15.163:0/2107973229 shutdown_connections 2024-07-26T06:13:42.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.481+0000 7fced0a00640 1 -- 172.21.15.163:0/2107973229 wait complete. 2024-07-26T06:13:42.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.484+0000 7fced0a00640 1 Processor -- start 2024-07-26T06:13:42.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.485+0000 7f6c5fe00640 1 -- 172.21.15.163:0/3098489741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c600b4420 msgr2=0x7f6c600b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.485+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/3098489741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c600b4420 0x7f6c600b2130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005fd0 tx=0x7f6c5802e780 comp rx=0 tx=0).stop 2024-07-26T06:13:42.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.485+0000 7fced0a00640 1 -- start start 2024-07-26T06:13:42.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.486+0000 7f6c5fe00640 1 -- 172.21.15.163:0/3098489741 shutdown_connections 2024-07-26T06:13:42.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.486+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/3098489741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c600b4420 0x7f6c600b2130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.486+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/3098489741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c600cbad0 0x7f6c60159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.486+0000 7f6c5fe00640 1 -- 172.21.15.163:0/3098489741 >> 172.21.15.163:0/3098489741 conn(0x7f6c600b0e30 msgr2=0x7f6c600b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.486+0000 7f6c5fe00640 1 -- 172.21.15.163:0/3098489741 shutdown_connections 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7f6c5fe00640 1 -- 172.21.15.163:0/3098489741 wait complete. 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7fced0a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec4039960 0x7fcec40cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7fced0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 0x7fcec404be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7fced0a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcec40e0a90 0x7fcec4033c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7fced0a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4006b10 con 0x7fcec40520e0 2024-07-26T06:13:42.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7fced0a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4006990 con 0x7fcec40e0a90 2024-07-26T06:13:42.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.487+0000 7fced0a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcec4006c90 con 0x7fcec4039960 2024-07-26T06:13:42.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.488+0000 7fcecaa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 0x7fcec404be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.488+0000 7fcecaa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 0x7fcec404be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45060/0 (socket says 172.21.15.163:45060) 2024-07-26T06:13:42.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.488+0000 7fcecaa00640 1 -- 172.21.15.163:0/3313502271 learned_addr learned my addr 172.21.15.163:0/3313502271 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.488+0000 7fcecbe00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcec40e0a90 0x7fcec4033c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.489+0000 7fcecb400640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec4039960 0x7fcec40cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.489+0000 7fcecbe00640 1 -- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec4039960 msgr2=0x7fcec40cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.489+0000 7f6c5fe00640 1 Processor -- start 2024-07-26T06:13:42.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.489+0000 7fcecbe00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec4039960 0x7fcec40cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.489+0000 7fcecbe00640 1 -- 172.21.15.163:0/3313502271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 msgr2=0x7fcec404be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.490+0000 7fcecbe00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 0x7fcec404be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.490+0000 7fcecbe00640 1 -- 172.21.15.163:0/3313502271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec40d9e30 con 0x7fcec40e0a90 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 -- 172.21.15.163:0/748307747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f415d100 msgr2=0x7fb1f415d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 --2- 172.21.15.163:0/748307747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f415d100 0x7fb1f415d510 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009830 tx=0x7fb1e402a5f0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 -- 172.21.15.163:0/748307747 shutdown_connections 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 --2- 172.21.15.163:0/748307747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f415d100 0x7fb1f415d510 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 -- 172.21.15.163:0/748307747 >> 172.21.15.163:0/748307747 conn(0x7fb1f40d36b0 msgr2=0x7fb1f40d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 -- 172.21.15.163:0/748307747 shutdown_connections 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7fb1f9800640 1 -- 172.21.15.163:0/748307747 wait complete. 2024-07-26T06:13:42.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.490+0000 7f6c5fe00640 1 -- start start 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7f6c5fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c600b4420 0x7f6c60145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7f6c5fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60159370 0x7f6c601ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7f6c5fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 0x7f6c60072190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7f6c5fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6006f2f0 con 0x7f6c60159370 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7f6c5fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6c600cd210 con 0x7f6c601efed0 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.491+0000 7f6c5fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6c6006f470 con 0x7f6c600b4420 2024-07-26T06:13:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.490+0000 7fcecaa00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 0x7fcec404be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:42.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.492+0000 7f6c5f400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 0x7f6c60072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.492+0000 7f6c5f400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 0x7f6c60072190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40678/0 (socket says 172.21.15.163:40678) 2024-07-26T06:13:42.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.492+0000 7f6c5f400640 1 -- 172.21.15.163:0/1716900497 learned_addr learned my addr 172.21.15.163:0/1716900497 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.492+0000 7f6c5ea00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c600b4420 0x7f6c60145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.493+0000 7f6c56000640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60159370 0x7f6c601ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.493+0000 7f6c5f400640 1 -- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c600b4420 msgr2=0x7f6c60145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.493+0000 7f6c5f400640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c600b4420 0x7f6c60145960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.493+0000 7f6c5f400640 1 -- 172.21.15.163:0/1716900497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60159370 msgr2=0x7f6c601ef990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.493+0000 7f6c5f400640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60159370 0x7f6c601ef990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.493+0000 7f6c5f400640 1 -- 172.21.15.163:0/1716900497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c601f09b0 con 0x7f6c601efed0 2024-07-26T06:13:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.494+0000 7fcecbe00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcec40e0a90 0x7fcec4033c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcecc06aaf0 tx=0x7fcecc069a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.494+0000 7fb1f9800640 1 Processor -- start 2024-07-26T06:13:42.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.494+0000 7f6c5f400640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 0x7f6c60072190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800d860 tx=0x7f6c5800cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7fb1f9800640 1 -- start start 2024-07-26T06:13:42.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004050 con 0x7f6c601efed0 2024-07-26T06:13:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc068de0 con 0x7fcec40e0a90 2024-07-26T06:13:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcecc069070 con 0x7fcec40e0a90 2024-07-26T06:13:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc096480 con 0x7fcec40e0a90 2024-07-26T06:13:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c58044070 con 0x7f6c601efed0 2024-07-26T06:13:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004050 con 0x7f6c601efed0 2024-07-26T06:13:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c60145ea0 con 0x7f6c601efed0 2024-07-26T06:13:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.496+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c601f06d0 con 0x7f6c601efed0 2024-07-26T06:13:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec4036a30 con 0x7fcec40e0a90 2024-07-26T06:13:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.495+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec40e1bd0 con 0x7fcec40e0a90 2024-07-26T06:13:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fb1f9800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 0x7fb1f4150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea8005290 con 0x7fcec40e0a90 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fb1f9800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1f40b6540 0x7fb1f40b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fb1f9800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1f415d100 0x7fb1f415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fb1f9800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f416cd40 con 0x7fb1f40b53b0 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fb1f9800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f416cbc0 con 0x7fb1f415d100 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.497+0000 7fb1f9800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb1f416cec0 con 0x7fb1f40b6540 2024-07-26T06:13:42.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.498+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 0x7fb1f4150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.498+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c60156ae0 con 0x7f6c601efed0 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.498+0000 7fb1f2a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 0x7fb1f4150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45092/0 (socket says 172.21.15.163:45092) 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.498+0000 7fb1f2a00640 1 -- 172.21.15.163:0/521569774 learned_addr learned my addr 172.21.15.163:0/521569774 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.498+0000 7fb1f2000640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1f40b6540 0x7fb1f40b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7fb1f2a00640 1 -- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1f40b6540 msgr2=0x7fb1f40b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7fb1f2a00640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1f40b6540 0x7fb1f40b7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7fb1f2a00640 1 -- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1f415d100 msgr2=0x7fb1f415ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7fb1f2a00640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1f415d100 0x7fb1f415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7fb1f2a00640 1 -- 172.21.15.163:0/521569774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f40ca5a0 con 0x7fb1f40b53b0 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7f28b6000640 1 -- 172.21.15.163:0/451174758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b00cc360 msgr2=0x7f28b00adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7f28b6000640 1 --2- 172.21.15.163:0/451174758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b00cc360 0x7f28b00adca0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f28b0068710 tx=0x7f28a002a4a0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f28b6000640 1 -- 172.21.15.163:0/451174758 shutdown_connections 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f28b6000640 1 --2- 172.21.15.163:0/451174758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b00cc360 0x7f28b00adca0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f28b6000640 1 --2- 172.21.15.163:0/451174758 >> v2:172.21.15.163:3300/0 conn(0x7f28b0146ef0 0x7f28b00cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f28b6000640 1 -- 172.21.15.163:0/451174758 >> 172.21.15.163:0/451174758 conn(0x7f28b006bf40 msgr2=0x7f28b006c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f28b6000640 1 -- 172.21.15.163:0/451174758 shutdown_connections 2024-07-26T06:13:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.499+0000 7fb1f2a00640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 0x7fb1f4150810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00c6e0 tx=0x7fb1dc00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0050d0 con 0x7fb1f40b53b0 2024-07-26T06:13:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1dc005270 con 0x7fb1f40b53b0 2024-07-26T06:13:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0035f0 con 0x7fb1f40b53b0 2024-07-26T06:13:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f40ca860 con 0x7fb1f40b53b0 2024-07-26T06:13:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.501+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcecc096620 con 0x7fcec40e0a90 2024-07-26T06:13:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f28b6000640 1 -- 172.21.15.163:0/451174758 wait complete. 2024-07-26T06:13:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.502+0000 7f28b6000640 1 Processor -- start 2024-07-26T06:13:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.501+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f40cae20 con 0x7fb1f40b53b0 2024-07-26T06:13:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.500+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6c5803e070 con 0x7f6c601efed0 2024-07-26T06:13:42.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.503+0000 7f28b6000640 1 -- start start 2024-07-26T06:13:42.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.503+0000 7fcebf400640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fceac0763a0 0x7fceac078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.504+0000 7f6c57400640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6c3c0764d0 0x7f6c3c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.503+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcecc1123d0 con 0x7fcec40e0a90 2024-07-26T06:13:42.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.504+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f415d510 con 0x7fb1f40b53b0 2024-07-26T06:13:42.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.504+0000 7fcecb400640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fceac0763a0 0x7fceac078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6c580bb730 con 0x7f6c601efed0 2024-07-26T06:13:42.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f6c5ea00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6c3c0764d0 0x7f6c3c078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f28b6000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28b00cc360 0x7f28b0071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f28b6000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b0146ef0 0x7f28b01e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f28b6000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 0x7f28b01e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f28b6000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00ae360 con 0x7f28b00cc360 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f28b6000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00ae1e0 con 0x7f28b0146ef0 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7f28b6000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f28b00ae4e0 con 0x7f28b015da90 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.506+0000 7f28afe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 0x7f28b01e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.506+0000 7f28afe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 0x7f28b01e50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50762/0 (socket says 172.21.15.163:50762) 2024-07-26T06:13:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.506+0000 7f28afe00640 1 -- 172.21.15.163:0/2256524958 learned_addr learned my addr 172.21.15.163:0/2256524958 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.506+0000 7f28af400640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28b00cc360 0x7f28b0071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.506+0000 7f28aea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b0146ef0 0x7f28b01e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.505+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1dc003790 con 0x7fb1f40b53b0 2024-07-26T06:13:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.506+0000 7fcecb400640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fceac0763a0 0x7fceac078860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcec0034000 tx=0x7fcec0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28af400640 1 -- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 msgr2=0x7f28b01e50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28af400640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 0x7f28b01e50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28af400640 1 -- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b0146ef0 msgr2=0x7f28b01e4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28af400640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b0146ef0 0x7f28b01e4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28af400640 1 -- 172.21.15.163:0/2256524958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b01e9060 con 0x7f28b00cc360 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28afe00640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 0x7f28b01e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28aea00640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b0146ef0 0x7f28b01e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:42.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7fb1eaa00640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb1d00762d0 0x7fb1d0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7f28af400640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28b00cc360 0x7f28b0071fa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f289c00e790 tx=0x7f289c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.507+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb1dc0972f0 con 0x7fb1f40b53b0 2024-07-26T06:13:42.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.508+0000 7fb1f2000640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb1d00762d0 0x7fb1d0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.509+0000 7f6c5ea00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6c3c0764d0 0x7f6c3c078990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c50000f60 tx=0x7f6c50009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.509+0000 7fb1f2000640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb1d00762d0 0x7fb1d0078790 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40b8bb0 tx=0x7fb1ec009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.514+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c00eea0 con 0x7f28b00cc360 2024-07-26T06:13:42.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.514+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f289c00cdb0 con 0x7f28b00cc360 2024-07-26T06:13:42.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.514+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c010690 con 0x7f28b00cc360 2024-07-26T06:13:42.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.515+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1f40ca3a0 con 0x7fb1f40b53b0 2024-07-26T06:13:42.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.514+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b00714c0 con 0x7f28b00cc360 2024-07-26T06:13:42.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.515+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b01e8d30 con 0x7f28b00cc360 2024-07-26T06:13:42.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.516+0000 7f806aa00640 1 -- 172.21.15.163:0/1665222060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80641545e0 msgr2=0x7f806415b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.516+0000 7f806aa00640 1 --2- 172.21.15.163:0/1665222060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80641545e0 0x7f806415b830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f80640aeed0 tx=0x7f805002a5f0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcecc0db750 con 0x7fcec40e0a90 2024-07-26T06:13:42.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b00712c0 con 0x7f28b00cc360 2024-07-26T06:13:42.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f806aa00640 1 -- 172.21.15.163:0/1665222060 shutdown_connections 2024-07-26T06:13:42.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f806aa00640 1 --2- 172.21.15.163:0/1665222060 >> v2:172.21.15.164:3300/0 conn(0x7f806415b390 0x7f80640b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f806aa00640 1 --2- 172.21.15.163:0/1665222060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80641545e0 0x7f806415b830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f806aa00640 1 -- 172.21.15.163:0/1665222060 >> 172.21.15.163:0/1665222060 conn(0x7f806406d420 msgr2=0x7f806406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f806aa00640 1 -- 172.21.15.163:0/1665222060 shutdown_connections 2024-07-26T06:13:42.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.517+0000 7f806aa00640 1 -- 172.21.15.163:0/1665222060 wait complete. 2024-07-26T06:13:42.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.520+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f289c010830 con 0x7f28b00cc360 2024-07-26T06:13:42.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.521+0000 7f28a7400640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f28900762d0 0x7f2890078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.521+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f289c098040 con 0x7f28b00cc360 2024-07-26T06:13:42.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.520+0000 7f806aa00640 1 Processor -- start 2024-07-26T06:13:42.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f28aea00640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f28900762d0 0x7f2890078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.520+0000 7f806aa00640 1 -- start start 2024-07-26T06:13:42.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f806aa00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f806406bf40 0x7f8064151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f806aa00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f806415b390 0x7f80640ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f806aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 0x7f80640a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f806aa00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8064072f20 con 0x7f80640adec0 2024-07-26T06:13:42.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f806aa00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8064072da0 con 0x7f806415b390 2024-07-26T06:13:42.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.522+0000 7f806aa00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f80640730a0 con 0x7f806406bf40 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.523+0000 7f8063400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f806415b390 0x7f80640ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.523+0000 7f8068a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 0x7f80640a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.523+0000 7f8068a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 0x7f80640a9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45122/0 (socket says 172.21.15.163:45122) 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.523+0000 7f8068a00640 1 -- 172.21.15.163:0/3519607639 learned_addr learned my addr 172.21.15.163:0/3519607639 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.524+0000 7f8068a00640 1 -- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f806406bf40 msgr2=0x7f8064151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.524+0000 7f8068a00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f806406bf40 0x7f8064151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.524+0000 7f8068a00640 1 -- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f806415b390 msgr2=0x7f80640ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.524+0000 7f8068a00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f806415b390 0x7f80640ad980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.524+0000 7f8068a00640 1 -- 172.21.15.163:0/3519607639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80640b1cd0 con 0x7f80640adec0 2024-07-26T06:13:42.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.524+0000 7f8068a00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 0x7f80640a9ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f805400c3c0 tx=0x7f805400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.525+0000 7f28aea00640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f28900762d0 0x7f2890078790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28b0144150 tx=0x7f28a8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.527+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f289c061340 con 0x7f28b00cc360 2024-07-26T06:13:42.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.526+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054017070 con 0x7f80640adec0 2024-07-26T06:13:42.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.527+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80540053c0 con 0x7f80640adec0 2024-07-26T06:13:42.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.527+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054017070 con 0x7f80640adec0 2024-07-26T06:13:42.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.527+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80640ae4f0 con 0x7f80640adec0 2024-07-26T06:13:42.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.527+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8064212970 con 0x7f80640adec0 2024-07-26T06:13:42.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.530+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6c58085120 con 0x7f6c601efed0 2024-07-26T06:13:42.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.530+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8064212480 con 0x7f80640adec0 2024-07-26T06:13:42.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.532+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f805401a070 con 0x7f80640adec0 2024-07-26T06:13:42.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.533+0000 7f805be00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80440766d0 0x7f8044078b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.533+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80540050d0 con 0x7f80640adec0 2024-07-26T06:13:42.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.535+0000 7f8063e00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80440766d0 0x7f8044078b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.536+0000 7f8063e00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80440766d0 0x7f8044078b90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8050033000 tx=0x7f80500046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.542+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f805405f720 con 0x7f80640adec0 2024-07-26T06:13:42.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.567+0000 7f13d3200640 1 -- 172.21.15.163:0/2666641408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 msgr2=0x7f13cc036850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.567+0000 7f13d3200640 1 --2- 172.21.15.163:0/2666641408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc036850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f13c0005fd0 tx=0x7f13c002b130 comp rx=0 tx=0).stop 2024-07-26T06:13:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.568+0000 7f13d3200640 1 -- 172.21.15.163:0/2666641408 shutdown_connections 2024-07-26T06:13:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.568+0000 7f13d3200640 1 --2- 172.21.15.163:0/2666641408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc036850 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.568+0000 7f13d3200640 1 --2- 172.21.15.163:0/2666641408 >> v2:172.21.15.164:3300/0 conn(0x7f13cc036470 0x7f13cc00a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.568+0000 7f13d3200640 1 -- 172.21.15.163:0/2666641408 >> 172.21.15.163:0/2666641408 conn(0x7f13cc01dd80 msgr2=0x7f13cc0549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.568+0000 7f13d3200640 1 -- 172.21.15.163:0/2666641408 shutdown_connections 2024-07-26T06:13:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.568+0000 7f13d3200640 1 -- 172.21.15.163:0/2666641408 wait complete. 2024-07-26T06:13:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.571+0000 7f13d3200640 1 Processor -- start 2024-07-26T06:13:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.571+0000 7f13d3200640 1 -- start start 2024-07-26T06:13:42.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.573+0000 7f13d3200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc178c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7f13d3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc036470 0x7f13cc0d8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7fc638000640 1 -- 172.21.15.163:0/3988641025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 msgr2=0x7fc6300b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7f13d3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13cc04efb0 0x7f13cc174d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7fc638000640 1 --2- 172.21.15.163:0/3988641025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc6300b6e30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc628009be0 tx=0x7fc62802a870 comp rx=0 tx=0).stop 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7fc638000640 1 -- 172.21.15.163:0/3988641025 shutdown_connections 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7fc638000640 1 --2- 172.21.15.163:0/3988641025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc6300b6e30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7fc638000640 1 -- 172.21.15.163:0/3988641025 >> 172.21.15.163:0/3988641025 conn(0x7fc6300d36b0 msgr2=0x7fc6300d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.575+0000 7fc638000640 1 -- 172.21.15.163:0/3988641025 shutdown_connections 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.575+0000 7fc638000640 1 -- 172.21.15.163:0/3988641025 wait complete. 2024-07-26T06:13:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7f13d1e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc178c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.574+0000 7f13d1400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc036470 0x7f13cc0d8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.576+0000 7f13d1e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc178c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50780/0 (socket says 172.21.15.163:50780) 2024-07-26T06:13:42.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.576+0000 7f13d1e00640 1 -- 172.21.15.163:0/1534170564 learned_addr learned my addr 172.21.15.163:0/1534170564 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.577+0000 7fc638000640 1 Processor -- start 2024-07-26T06:13:42.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.577+0000 7fc638000640 1 -- start start 2024-07-26T06:13:42.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.575+0000 7f13d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13cc04efb0 0x7f13cc174d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.579+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc004b00 con 0x7f13cc04efb0 2024-07-26T06:13:42.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.579+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc007560 con 0x7f13cc036470 2024-07-26T06:13:42.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.579+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f13cc004c80 con 0x7f13cc00bbc0 2024-07-26T06:13:42.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.579+0000 7fc638000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 0x7fc63015af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7f13d2800640 1 -- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 msgr2=0x7f13cc178c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7f13d2800640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc178c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7f13d2800640 1 -- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc036470 msgr2=0x7f13cc0d8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7f13d2800640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc036470 0x7f13cc0d8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7f13d2800640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13cc175c80 con 0x7f13cc04efb0 2024-07-26T06:13:42.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7fc638000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc630156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7fc635600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 0x7fc63015af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7fc635600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 0x7fc63015af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45132/0 (socket says 172.21.15.163:45132) 2024-07-26T06:13:42.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7fc635600640 1 -- 172.21.15.163:0/1130038814 learned_addr learned my addr 172.21.15.163:0/1130038814 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc63015d650 0x7fc630157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.580+0000 7f13d2800640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13cc04efb0 0x7f13cc174d50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13c0031000 tx=0x7f13c0008a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300b7370 con 0x7fc6300b53b0 2024-07-26T06:13:42.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300b74f0 con 0x7fc63015d650 2024-07-26T06:13:42.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc6300b7670 con 0x7fc6300b5800 2024-07-26T06:13:42.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7fc634c00640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc630156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0040070 con 0x7f13cc04efb0 2024-07-26T06:13:42.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13c003a570 con 0x7f13cc04efb0 2024-07-26T06:13:42.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13cc1756c0 con 0x7f13cc04efb0 2024-07-26T06:13:42.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0005d50 con 0x7f13cc04efb0 2024-07-26T06:13:42.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13cc178470 con 0x7f13cc04efb0 2024-07-26T06:13:42.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.581+0000 7fc636000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc63015d650 0x7fc630157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7fc635600640 1 -- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 msgr2=0x7fc630156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13cc0503e0 con 0x7f13cc04efb0 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7fc635600640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc630156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7fc635600640 1 -- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc63015d650 msgr2=0x7fc630157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7fc635600640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc63015d650 0x7fc630157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7fc635600640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63015fb50 con 0x7fc6300b53b0 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.584+0000 7fc634c00640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc630156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.585+0000 7fc635600640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 0x7fc63015af90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00b3f0 tx=0x7fc61c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.586+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c015020 con 0x7fc6300b53b0 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.586+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc61c011940 con 0x7fc6300b53b0 2024-07-26T06:13:42.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.586+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c011d90 con 0x7fc6300b53b0 2024-07-26T06:13:42.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.586+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63015fe30 con 0x7fc6300b53b0 2024-07-26T06:13:42.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.586+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6301603f0 con 0x7fc6300b53b0 2024-07-26T06:13:42.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.588+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f13c002bb30 con 0x7f13cc04efb0 2024-07-26T06:13:42.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.589+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c005290 con 0x7fc6300b53b0 2024-07-26T06:13:42.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.591+0000 7f13ca000640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13b40762d0 0x7f13b4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.591+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f13c00bb550 con 0x7f13cc04efb0 2024-07-26T06:13:42.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.592+0000 7f13d1e00640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13b40762d0 0x7f13b4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.593+0000 7f13d1e00640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13b40762d0 0x7f13b4078790 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f13d406b090 tx=0x7f13d406b730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.602+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13c00847d0 con 0x7f13cc04efb0 2024-07-26T06:13:42.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.605+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc61c018020 con 0x7fc6300b53b0 2024-07-26T06:13:42.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.610+0000 7fc62d600640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6100763a0 0x7fc610078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.610+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc61c097bf0 con 0x7fc6300b53b0 2024-07-26T06:13:42.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.610+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc61c0c6b30 con 0x7fc6300b53b0 2024-07-26T06:13:42.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.611+0000 7fc634c00640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6100763a0 0x7fc610078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.616+0000 7fc634c00640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6100763a0 0x7fc610078860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.627+0000 7ff826200640 1 -- 172.21.15.163:0/3146077152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cbec0 msgr2=0x7ff8200cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.627+0000 7ff826200640 1 --2- 172.21.15.163:0/3146077152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cbec0 0x7ff8200cde90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009830 tx=0x7ff80c02a360 comp rx=0 tx=0).stop 2024-07-26T06:13:42.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.627+0000 7ff826200640 1 -- 172.21.15.163:0/3146077152 shutdown_connections 2024-07-26T06:13:42.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.628+0000 7ff826200640 1 --2- 172.21.15.163:0/3146077152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff82015dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.628+0000 7ff826200640 1 --2- 172.21.15.163:0/3146077152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cbec0 0x7ff8200cde90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.628+0000 7ff826200640 1 -- 172.21.15.163:0/3146077152 >> 172.21.15.163:0/3146077152 conn(0x7ff82006bf40 msgr2=0x7ff82006c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.628+0000 7ff826200640 1 -- 172.21.15.163:0/3146077152 shutdown_connections 2024-07-26T06:13:42.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.628+0000 7ff826200640 1 -- 172.21.15.163:0/3146077152 wait complete. 2024-07-26T06:13:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.632+0000 7ff826200640 1 Processor -- start 2024-07-26T06:13:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.632+0000 7ff826200640 1 -- start start 2024-07-26T06:13:42.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.634+0000 7ff826200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff8200725d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.635+0000 7ff826200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8200cbec0 0x7ff82006e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.635+0000 7ff81f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff8200725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.635+0000 7ff826200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cc2a0 0x7ff82006ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.635+0000 7ff826200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff820162050 con 0x7ff8200cbec0 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.635+0000 7ff826200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff820161ed0 con 0x7ff8200cc2a0 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.635+0000 7ff826200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff8201621d0 con 0x7ff8200b22d0 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.636+0000 7ff81f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff8200725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50804/0 (socket says 172.21.15.163:50804) 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.636+0000 7ff81f400640 1 -- 172.21.15.163:0/4290043914 learned_addr learned my addr 172.21.15.163:0/4290043914 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.636+0000 7ff81ea00640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8200cbec0 0x7ff82006e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.636+0000 7ff81fe00640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cc2a0 0x7ff82006ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.637+0000 7ff81f400640 1 -- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cc2a0 msgr2=0x7ff82006ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.637+0000 7ff81f400640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cc2a0 0x7ff82006ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.637+0000 7ff81f400640 1 -- 172.21.15.163:0/4290043914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8200cbec0 msgr2=0x7ff82006e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.637+0000 7ff81f400640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8200cbec0 0x7ff82006e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.637+0000 7ff81f400640 1 -- 172.21.15.163:0/4290043914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff820071530 con 0x7ff8200b22d0 2024-07-26T06:13:42.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.637+0000 7ff81ea00640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8200cbec0 0x7ff82006e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:42.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.638+0000 7ff81f400640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff8200725d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009780 tx=0x7ff80c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.638+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c03f020 con 0x7ff8200b22d0 2024-07-26T06:13:42.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.638+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff80c033e10 con 0x7ff8200b22d0 2024-07-26T06:13:42.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.638+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c037040 con 0x7ff8200b22d0 2024-07-26T06:13:42.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.638+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff820071810 con 0x7ff8200b22d0 2024-07-26T06:13:42.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.639+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff820071dd0 con 0x7ff8200b22d0 2024-07-26T06:13:42.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.640+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8200712a0 con 0x7ff8200b22d0 2024-07-26T06:13:42.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.644+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff80c042020 con 0x7ff8200b22d0 2024-07-26T06:13:42.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.646+0000 7ff817400640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff8000762d0 0x7ff800078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:42.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.646+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff80c0b80d0 con 0x7ff8200b22d0 2024-07-26T06:13:42.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.647+0000 7ff81ea00640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff8000762d0 0x7ff800078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:42.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.648+0000 7ff81ea00640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff8000762d0 0x7ff800078790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff8200ce3a0 tx=0x7ff810005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:42.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.659+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff80c049030 con 0x7ff8200b22d0 2024-07-26T06:13:42.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.937+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb1f40d1b60 con 0x7fb1f40b53b0 2024-07-26T06:13:42.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.939+0000 7fb1eaa00640 1 -- 172.21.15.163:0/521569774 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb1dc060c40 con 0x7fb1f40b53b0 2024-07-26T06:13:42.941 INFO:teuthology.orchestra.run.smithi163.stdout:64424509446 2024-07-26T06:13:42.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.943+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb1d00762d0 msgr2=0x7fb1d0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.943+0000 7fb1f9800640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb1d00762d0 0x7fb1d0078790 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40b8bb0 tx=0x7fb1ec009040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.943+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 msgr2=0x7fb1f4150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.943+0000 7fb1f9800640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 0x7fb1f4150810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00c6e0 tx=0x7fb1dc00cba0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.944+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 shutdown_connections 2024-07-26T06:13:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.944+0000 7fb1f9800640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb1d00762d0 0x7fb1d0078790 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.944+0000 7fb1f9800640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1f415d100 0x7fb1f415ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.944+0000 7fb1f9800640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1f40b6540 0x7fb1f40b7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.944+0000 7fb1f9800640 1 --2- 172.21.15.163:0/521569774 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1f40b53b0 0x7fb1f4150810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.944+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 >> 172.21.15.163:0/521569774 conn(0x7fb1f40d36b0 msgr2=0x7fb1f40d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.945+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 shutdown_connections 2024-07-26T06:13:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.945+0000 7fb1f9800640 1 -- 172.21.15.163:0/521569774 wait complete. 2024-07-26T06:13:42.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.951+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcea8005850 con 0x7fcec40e0a90 2024-07-26T06:13:42.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.952+0000 7fcebf400640 1 -- 172.21.15.163:0/3313502271 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcecc0dfa50 con 0x7fcec40e0a90 2024-07-26T06:13:42.956 INFO:teuthology.orchestra.run.smithi163.stdout:60129542151 2024-07-26T06:13:42.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.956+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fceac0763a0 msgr2=0x7fceac078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.956+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6c60156280 con 0x7f6c601efed0 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.956+0000 7fced0a00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fceac0763a0 0x7fceac078860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcec0034000 tx=0x7fcec0004730 comp rx=0 tx=0).stop 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.956+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcec40e0a90 msgr2=0x7fcec4033c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.956+0000 7fced0a00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcec40e0a90 0x7fcec4033c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcecc06aaf0 tx=0x7fcecc069a50 comp rx=0 tx=0).stop 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 shutdown_connections 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7fced0a00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fceac0763a0 0x7fceac078860 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7fced0a00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcec40e0a90 0x7fcec4033c50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7fced0a00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec40520e0 0x7fcec404be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7f6c57400640 1 -- 172.21.15.163:0/1716900497 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6c5808d110 con 0x7f6c601efed0 2024-07-26T06:13:42.960 INFO:teuthology.orchestra.run.smithi163.stdout:60129542150 2024-07-26T06:13:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7fced0a00640 1 --2- 172.21.15.163:0/3313502271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcec4039960 0x7fcec40cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.957+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 >> 172.21.15.163:0/3313502271 conn(0x7fcec401d930 msgr2=0x7fcec4039250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.958+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 shutdown_connections 2024-07-26T06:13:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.958+0000 7fced0a00640 1 -- 172.21.15.163:0/3313502271 wait complete. 2024-07-26T06:13:42.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6c3c0764d0 msgr2=0x7f6c3c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6c3c0764d0 0x7f6c3c078990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c50000f60 tx=0x7f6c50009040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 msgr2=0x7f6c60072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 0x7f6c60072190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800d860 tx=0x7f6c5800cb40 comp rx=0 tx=0).stop 2024-07-26T06:13:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 shutdown_connections 2024-07-26T06:13:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6c3c0764d0 0x7f6c3c078990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.961+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c601efed0 0x7f6c60072190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.962+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60159370 0x7f6c601ef990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.962+0000 7f6c5fe00640 1 --2- 172.21.15.163:0/1716900497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c600b4420 0x7f6c60145960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.962+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 >> 172.21.15.163:0/1716900497 conn(0x7f6c600b0e30 msgr2=0x7f6c600aed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.962+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 shutdown_connections 2024-07-26T06:13:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.962+0000 7f6c5fe00640 1 -- 172.21.15.163:0/1716900497 wait complete. 2024-07-26T06:13:42.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.0 2024-07-26T06:13:42.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.983+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f28b0141ba0 con 0x7f28b00cc360 2024-07-26T06:13:42.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.983+0000 7f28a7400640 1 -- 172.21.15.163:0/2256524958 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f289c065640 con 0x7f28b00cc360 2024-07-26T06:13:42.984 INFO:teuthology.orchestra.run.smithi163.stdout:64424509447 2024-07-26T06:13:42.986 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2024-07-26T06:13:42.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f28900762d0 msgr2=0x7f2890078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f28900762d0 0x7f2890078790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28b0144150 tx=0x7f28a8009040 comp rx=0 tx=0).stop 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28b00cc360 msgr2=0x7f28b0071fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28b00cc360 0x7f28b0071fa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f289c00e790 tx=0x7f289c00ec50 comp rx=0 tx=0).stop 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 shutdown_connections 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f28900762d0 0x7f2890078790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28b015da90 0x7f28b01e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28b0146ef0 0x7f28b01e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 --2- 172.21.15.163:0/2256524958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28b00cc360 0x7f28b0071fa0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 >> 172.21.15.163:0/2256524958 conn(0x7f28b006bf40 msgr2=0x7f28b01454f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 shutdown_connections 2024-07-26T06:13:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.987+0000 7f28b6000640 1 -- 172.21.15.163:0/2256524958 wait complete. 2024-07-26T06:13:42.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.4 2024-07-26T06:13:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.993+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f806415b830 con 0x7f80640adec0 2024-07-26T06:13:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.993+0000 7f805be00640 1 -- 172.21.15.163:0/3519607639 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8054063a20 con 0x7f80640adec0 2024-07-26T06:13:42.995 INFO:teuthology.orchestra.run.smithi163.stdout:60129542149 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.997+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80440766d0 msgr2=0x7f8044078b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.997+0000 7f806aa00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80440766d0 0x7f8044078b90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8050033000 tx=0x7f80500046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.997+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 msgr2=0x7f80640a9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.997+0000 7f806aa00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 0x7f80640a9ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f805400c3c0 tx=0x7f805400c880 comp rx=0 tx=0).stop 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 shutdown_connections 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80440766d0 0x7f8044078b90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80640adec0 0x7f80640a9ee0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f806415b390 0x7f80640ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 --2- 172.21.15.163:0/3519607639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f806406bf40 0x7f8064151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 >> 172.21.15.163:0/3519607639 conn(0x7f806406d420 msgr2=0x7f8064158290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 shutdown_connections 2024-07-26T06:13:42.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:42.998+0000 7f806aa00640 1 -- 172.21.15.163:0/3519607639 wait complete. 2024-07-26T06:13:43.014 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.3 2024-07-26T06:13:43.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.2 2024-07-26T06:13:43.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.017+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc60c005850 con 0x7fc6300b53b0 2024-07-26T06:13:43.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.018+0000 7fc62d600640 1 -- 172.21.15.163:0/1130038814 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc61c061530 con 0x7fc6300b53b0 2024-07-26T06:13:43.019 INFO:teuthology.orchestra.run.smithi163.stdout:64424509445 2024-07-26T06:13:43.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6100763a0 msgr2=0x7fc610078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:43.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6100763a0 0x7fc610078860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).stop 2024-07-26T06:13:43.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 msgr2=0x7fc63015af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:43.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 0x7fc63015af90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00b3f0 tx=0x7fc61c00b8b0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 shutdown_connections 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6100763a0 0x7fc610078860 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc63015d650 0x7fc630157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6300b5800 0x7fc630156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 --2- 172.21.15.163:0/1130038814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6300b53b0 0x7fc63015af90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 >> 172.21.15.163:0/1130038814 conn(0x7fc6300d36b0 msgr2=0x7fc6300b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 shutdown_connections 2024-07-26T06:13:43.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.020+0000 7fc638000640 1 -- 172.21.15.163:0/1130038814 wait complete. 2024-07-26T06:13:43.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.025+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f13cc1791b0 con 0x7f13cc04efb0 2024-07-26T06:13:43.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.026+0000 7f13ca000640 1 -- 172.21.15.163:0/1534170564 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f13c0088ad0 con 0x7f13cc04efb0 2024-07-26T06:13:43.027 INFO:teuthology.orchestra.run.smithi163.stdout:64424509446 2024-07-26T06:13:43.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13b40762d0 msgr2=0x7f13b4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:43.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13b40762d0 0x7f13b4078790 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f13d406b090 tx=0x7f13d406b730 comp rx=0 tx=0).stop 2024-07-26T06:13:43.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13cc04efb0 msgr2=0x7f13cc174d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:43.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13cc04efb0 0x7f13cc174d50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13c0031000 tx=0x7f13c0008a60 comp rx=0 tx=0).stop 2024-07-26T06:13:43.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 shutdown_connections 2024-07-26T06:13:43.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13b40762d0 0x7f13b4078790 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13cc04efb0 0x7f13cc174d50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc036470 0x7f13cc0d8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 --2- 172.21.15.163:0/1534170564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc00bbc0 0x7f13cc178c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 >> 172.21.15.163:0/1534170564 conn(0x7f13cc01dd80 msgr2=0x7f13cc04f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:43.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 shutdown_connections 2024-07-26T06:13:43.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.028+0000 7f13d3200640 1 -- 172.21.15.163:0/1534170564 wait complete. 2024-07-26T06:13:43.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.5 2024-07-26T06:13:43.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.041+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff8200b2dd0 con 0x7ff8200b22d0 2024-07-26T06:13:43.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.041+0000 7ff817400640 1 -- 172.21.15.163:0/4290043914 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff80c081980 con 0x7ff8200b22d0 2024-07-26T06:13:43.042 INFO:teuthology.orchestra.run.smithi163.stdout:64424509445 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff8000762d0 msgr2=0x7ff800078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff8000762d0 0x7ff800078790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff8200ce3a0 tx=0x7ff810005f90 comp rx=0 tx=0).stop 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 msgr2=0x7ff8200725d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff8200725d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009780 tx=0x7ff80c033c20 comp rx=0 tx=0).stop 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 shutdown_connections 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff8000762d0 0x7ff800078790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8200cc2a0 0x7ff82006ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8200cbec0 0x7ff82006e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 --2- 172.21.15.163:0/4290043914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8200b22d0 0x7ff8200725d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:43.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 >> 172.21.15.163:0/4290043914 conn(0x7ff82006bf40 msgr2=0x7ff82015ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:43.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 shutdown_connections 2024-07-26T06:13:43.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:43.044+0000 7ff826200640 1 -- 172.21.15.163:0/4290043914 wait complete. 2024-07-26T06:13:43.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.6 2024-07-26T06:13:43.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.7 2024-07-26T06:13:43.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-07-26T06:13:43.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-07-26T06:13:43.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-07-26T06:13:44.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-07-26T06:13:44.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-07-26T06:13:44.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-07-26T06:13:44.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-07-26T06:13:44.063 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.243+0000 7f16d9400640 1 -- 172.21.15.163:0/3971052427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 msgr2=0x7f16d40cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.243+0000 7f16d9400640 1 --2- 172.21.15.163:0/3971052427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 0x7f16d40cd910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009830 tx=0x7f16c002a480 comp rx=0 tx=0).stop 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.244+0000 7f16d9400640 1 -- 172.21.15.163:0/3971052427 shutdown_connections 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.244+0000 7f16d9400640 1 --2- 172.21.15.163:0/3971052427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 0x7f16d40cd910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.244+0000 7f16d9400640 1 --2- 172.21.15.163:0/3971052427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b4d40 0x7f16d40b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.244+0000 7f16d9400640 1 -- 172.21.15.163:0/3971052427 >> 172.21.15.163:0/3971052427 conn(0x7f16d40cfd40 msgr2=0x7f16d40d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.244+0000 7f16d9400640 1 -- 172.21.15.163:0/3971052427 shutdown_connections 2024-07-26T06:13:44.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.244+0000 7f16d9400640 1 -- 172.21.15.163:0/3971052427 wait complete. 2024-07-26T06:13:44.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.247+0000 7f16d9400640 1 Processor -- start 2024-07-26T06:13:44.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.247+0000 7f16d9400640 1 -- start start 2024-07-26T06:13:44.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.249+0000 7f16d9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d40b4d40 0x7f16d40cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.250+0000 7f16d9400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 0x7f16d40cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.250+0000 7f16d9400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b5a80 0x7f16d40cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.250+0000 7f16d9400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f16d40d1a20 con 0x7f16d40b4d40 2024-07-26T06:13:44.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.251+0000 7f16d8a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b5a80 0x7f16d40cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.251+0000 7f16d3400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 0x7f16d40cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.251+0000 7f16d8a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b5a80 0x7f16d40cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50824/0 (socket says 172.21.15.163:50824) 2024-07-26T06:13:44.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.251+0000 7f16d8a00640 1 -- 172.21.15.163:0/2770012752 learned_addr learned my addr 172.21.15.163:0/2770012752 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.251+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f16d40d18a0 con 0x7f16d40b5660 2024-07-26T06:13:44.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.251+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f16d40ce560 con 0x7f16d40b5a80 2024-07-26T06:13:44.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.252+0000 7f16d3e00640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d40b4d40 0x7f16d40cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.253+0000 7f16d3e00640 1 -- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b5a80 msgr2=0x7f16d40cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.253+0000 7f16d3e00640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b5a80 0x7f16d40cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.253+0000 7f16d3e00640 1 -- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 msgr2=0x7f16d40cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.253+0000 7f16d3e00640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 0x7f16d40cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.253+0000 7f16d3e00640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d40ac580 con 0x7f16d40b4d40 2024-07-26T06:13:44.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.254+0000 7f16d3e00640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d40b4d40 0x7f16d40cb6c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00c6a0 tx=0x7f16cc00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.255+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc01b070 con 0x7f16d40b4d40 2024-07-26T06:13:44.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.255+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d40ac860 con 0x7f16d40b4d40 2024-07-26T06:13:44.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.255+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d40ace20 con 0x7f16d40b4d40 2024-07-26T06:13:44.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.256+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16cc00cd80 con 0x7f16d40b4d40 2024-07-26T06:13:44.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.257+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc01b070 con 0x7f16d40b4d40 2024-07-26T06:13:44.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.259+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16ac005290 con 0x7f16d40b4d40 2024-07-26T06:13:44.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.270+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f16cc01e070 con 0x7f16d40b4d40 2024-07-26T06:13:44.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.272+0000 7f16cbe00640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f16b40764d0 0x7f16b4078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.272+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f16cc097a70 con 0x7f16d40b4d40 2024-07-26T06:13:44.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.272+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f16cc09b280 con 0x7f16d40b4d40 2024-07-26T06:13:44.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.279+0000 7f16d3400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f16b40764d0 0x7f16b4078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.283+0000 7f16d3400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f16b40764d0 0x7f16b4078990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16c0034000 tx=0x7f16c0004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.290+0000 7fb600a00640 1 -- 172.21.15.163:0/1967388418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0ce830 msgr2=0x7fb5fc0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.290+0000 7fb600a00640 1 --2- 172.21.15.163:0/1967388418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0ce830 0x7fb5fc0b64c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0690b0 tx=0x7fb5ec02a4b0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.291+0000 7fb600a00640 1 -- 172.21.15.163:0/1967388418 shutdown_connections 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.291+0000 7fb600a00640 1 --2- 172.21.15.163:0/1967388418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0ce830 0x7fb5fc0b64c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.291+0000 7fb600a00640 1 --2- 172.21.15.163:0/1967388418 >> v2:172.21.15.164:3300/0 conn(0x7fb5fc15d5a0 0x7fb5fc0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.291+0000 7fb600a00640 1 -- 172.21.15.163:0/1967388418 >> 172.21.15.163:0/1967388418 conn(0x7fb5fc0cfda0 msgr2=0x7fb5fc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.291+0000 7fb600a00640 1 -- 172.21.15.163:0/1967388418 shutdown_connections 2024-07-26T06:13:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.291+0000 7fb600a00640 1 -- 172.21.15.163:0/1967388418 wait complete. 2024-07-26T06:13:44.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.290+0000 7f43f6200640 1 -- 172.21.15.163:0/1815425549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 msgr2=0x7f43f00b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.290+0000 7f43f6200640 1 --2- 172.21.15.163:0/1815425549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f00b2ae0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43f00aed70 tx=0x7f43dc02c710 comp rx=0 tx=0).stop 2024-07-26T06:13:44.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.293+0000 7fb600a00640 1 Processor -- start 2024-07-26T06:13:44.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.293+0000 7fb600a00640 1 -- start start 2024-07-26T06:13:44.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.294+0000 7f43f6200640 1 -- 172.21.15.163:0/1815425549 shutdown_connections 2024-07-26T06:13:44.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.294+0000 7f43f6200640 1 --2- 172.21.15.163:0/1815425549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f00b2ae0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.294+0000 7f43f6200640 1 --2- 172.21.15.163:0/1815425549 >> v2:172.21.15.164:3300/0 conn(0x7f43f016c410 0x7f43f00b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.294+0000 7f43f6200640 1 -- 172.21.15.163:0/1815425549 >> 172.21.15.163:0/1815425549 conn(0x7f43f006d280 msgr2=0x7f43f006d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.294+0000 7f43f6200640 1 -- 172.21.15.163:0/1815425549 shutdown_connections 2024-07-26T06:13:44.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.294+0000 7f43f6200640 1 -- 172.21.15.163:0/1815425549 wait complete. 2024-07-26T06:13:44.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb600a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 0x7fb5fc156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb600a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5fc0ce830 0x7fb5fc0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb600a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5fc15d5a0 0x7fb5fc0c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb600a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc0b5170 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb600a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc0b4ff0 con 0x7fb5fc15d5a0 2024-07-26T06:13:44.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb600a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb5fc0b52f0 con 0x7fb5fc0ce830 2024-07-26T06:13:44.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb5fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 0x7fb5fc156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb5fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 0x7fb5fc156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45184/0 (socket says 172.21.15.163:45184) 2024-07-26T06:13:44.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.296+0000 7fb5fb400640 1 -- 172.21.15.163:0/282580669 learned_addr learned my addr 172.21.15.163:0/282580669 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7f43f6200640 1 Processor -- start 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7fb5f2a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5fc0ce830 0x7fb5fc0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7fb5fb400640 1 -- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5fc0ce830 msgr2=0x7fb5fc0c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7fb5fb400640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5fc0ce830 0x7fb5fc0c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7fb5fb400640 1 -- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5fc15d5a0 msgr2=0x7fb5fc0c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7fb5fb400640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5fc15d5a0 0x7fb5fc0c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.297+0000 7fb5fb400640 1 -- 172.21.15.163:0/282580669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5fc0c9500 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.298+0000 7f43f6200640 1 -- start start 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.298+0000 7fb5fb400640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 0x7fb5fc156ce0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400e7c0 tx=0x7fb5e400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.298+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400c8b0 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.299+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5fc0c97e0 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.299+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5fc0c91f0 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.299+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5e400cb40 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.299+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4005020 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.299+0000 7f43f6200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f0209130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.301+0000 7f43f6200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43f00cba90 0x7f43f0168da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.301+0000 7f43f6200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43f016c410 0x7f43f01696d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.301+0000 7f43f4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f0209130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7f43f6200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43f016a6b0 con 0x7f43f00b2670 2024-07-26T06:13:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7f43f6200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43f00cd280 con 0x7f43f00cba90 2024-07-26T06:13:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7f43f6200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f43f00b3020 con 0x7f43f016c410 2024-07-26T06:13:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7f43efe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43f00cba90 0x7f43f0168da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7f43f4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f0209130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45186/0 (socket says 172.21.15.163:45186) 2024-07-26T06:13:44.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7f43f4e00640 1 -- 172.21.15.163:0/729302955 learned_addr learned my addr 172.21.15.163:0/729302955 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.302+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb5e40051c0 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.303+0000 7f43f4e00640 1 -- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43f016c410 msgr2=0x7f43f01696d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.303+0000 7f43f4e00640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43f016c410 0x7f43f01696d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.303+0000 7f43f4e00640 1 -- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43f00cba90 msgr2=0x7f43f0168da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.303+0000 7f43f4e00640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43f00cba90 0x7f43f0168da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.303+0000 7f43f4e00640 1 -- 172.21.15.163:0/729302955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f00716c0 con 0x7f43f00b2670 2024-07-26T06:13:44.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.304+0000 7f43f4e00640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f0209130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7a0 tx=0x7f43e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.304+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d4005290 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.304+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8017070 con 0x7f43f00b2670 2024-07-26T06:13:44.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.305+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f00719a0 con 0x7f43f00b2670 2024-07-26T06:13:44.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.306+0000 7fb5f3e00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb5d80764d0 0x7fb5d8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.306+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb5e400cd10 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.305+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f0071f60 con 0x7f43f00b2670 2024-07-26T06:13:44.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.307+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43e8005070 con 0x7f43f00b2670 2024-07-26T06:13:44.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.307+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8017070 con 0x7f43f00b2670 2024-07-26T06:13:44.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.310+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f006a1f0 con 0x7f43f00b2670 2024-07-26T06:13:44.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.312+0000 7fb5f2a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb5d80764d0 0x7fb5d8078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.317+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5e40695a0 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.318+0000 7fb5f2a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb5d80764d0 0x7fb5d8078990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec034000 tx=0x7fb5ec004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.320+0000 7f5ac3200640 1 -- 172.21.15.163:0/927501695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc0e28f0 msgr2=0x7f5abc053080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.320+0000 7f5ac3200640 1 --2- 172.21.15.163:0/927501695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc0e28f0 0x7f5abc053080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5abc054410 tx=0x7f5ab402a560 comp rx=0 tx=0).stop 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.320+0000 7f5ac3200640 1 -- 172.21.15.163:0/927501695 shutdown_connections 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.320+0000 7f5ac3200640 1 --2- 172.21.15.163:0/927501695 >> v2:172.21.15.164:3300/0 conn(0x7f5abc057830 0x7f5abc0066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.320+0000 7f5ac3200640 1 --2- 172.21.15.163:0/927501695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc0e28f0 0x7f5abc053080 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.320+0000 7f5ac3200640 1 -- 172.21.15.163:0/927501695 >> 172.21.15.163:0/927501695 conn(0x7f5abc03a520 msgr2=0x7f5abc03a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.321+0000 7f5ac3200640 1 -- 172.21.15.163:0/927501695 shutdown_connections 2024-07-26T06:13:44.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.319+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f43e801a070 con 0x7f43f00b2670 2024-07-26T06:13:44.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.321+0000 7f43ecc00640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43d0076500 0x7f43d00789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.321+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f43e80970e0 con 0x7f43f00b2670 2024-07-26T06:13:44.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.322+0000 7f43efe00640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43d0076500 0x7f43d00789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.323+0000 7f5ac3200640 1 -- 172.21.15.163:0/927501695 wait complete. 2024-07-26T06:13:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.324+0000 7f43efe00640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43d0076500 0x7f43d00789c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f43f0168a60 tx=0x7f43e000b2e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.325+0000 7f5ac3200640 1 Processor -- start 2024-07-26T06:13:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.325+0000 7f5ac3200640 1 -- start start 2024-07-26T06:13:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.326+0000 7f5ac3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 0x7f5abc0e3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.326+0000 7f5ac3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc057830 0x7f5abc0d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.326+0000 7f5ac3200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5abc0e28f0 0x7f5abc0e4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.326+0000 7f5ac3200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc006bf0 con 0x7f5abc057830 2024-07-26T06:13:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.326+0000 7f5ac3200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc0e2ee0 con 0x7f5abc03bc60 2024-07-26T06:13:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.326+0000 7f5ac3200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5abc006d70 con 0x7f5abc0e28f0 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.327+0000 7f5ac1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc057830 0x7f5abc0d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.327+0000 7f5ac1e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 0x7f5abc0e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.327+0000 7f5ac1e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 0x7f5abc0e3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40842/0 (socket says 172.21.15.163:40842) 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.327+0000 7f5ac1e00640 1 -- 172.21.15.163:0/711480920 learned_addr learned my addr 172.21.15.163:0/711480920 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7f5ac1e00640 1 -- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5abc0e28f0 msgr2=0x7f5abc0e4950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7f5ac1e00640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5abc0e28f0 0x7f5abc0e4950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7f5ac1e00640 1 -- 172.21.15.163:0/711480920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc057830 msgr2=0x7f5abc0d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7fae05000640 1 -- 172.21.15.163:0/903187643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf80e28f0 msgr2=0x7fadf8053080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7fae05000640 1 --2- 172.21.15.163:0/903187643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf80e28f0 0x7fadf8053080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005fd0 tx=0x7fadf402a610 comp rx=0 tx=0).stop 2024-07-26T06:13:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43f006a1f0 con 0x7f43f00b2670 2024-07-26T06:13:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7fae05000640 1 -- 172.21.15.163:0/903187643 shutdown_connections 2024-07-26T06:13:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7fae05000640 1 --2- 172.21.15.163:0/903187643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadf8057830 0x7fadf80066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7fae05000640 1 --2- 172.21.15.163:0/903187643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf80e28f0 0x7fadf8053080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.329+0000 7f5ac1e00640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc057830 0x7f5abc0d2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.329+0000 7f5ac1e00640 1 -- 172.21.15.163:0/711480920 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5abc0ded00 con 0x7f5abc03bc60 2024-07-26T06:13:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.328+0000 7fae05000640 1 -- 172.21.15.163:0/903187643 >> 172.21.15.163:0/903187643 conn(0x7fadf803a520 msgr2=0x7fadf803a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.329+0000 7fae05000640 1 -- 172.21.15.163:0/903187643 shutdown_connections 2024-07-26T06:13:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.329+0000 7f5ac1400640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc057830 0x7f5abc0d2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.329+0000 7fae05000640 1 -- 172.21.15.163:0/903187643 wait complete. 2024-07-26T06:13:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.330+0000 7f5ac1e00640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 0x7f5abc0e3060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4034040 tx=0x7f5ab4003910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.330+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab403e030 con 0x7f5abc03bc60 2024-07-26T06:13:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.330+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ab4003e00 con 0x7f5abc03bc60 2024-07-26T06:13:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.330+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4035040 con 0x7f5abc03bc60 2024-07-26T06:13:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.331+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5abc0defe0 con 0x7f5abc03bc60 2024-07-26T06:13:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.331+0000 7fae05000640 1 Processor -- start 2024-07-26T06:13:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.332+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5abc0d3440 con 0x7f5abc03bc60 2024-07-26T06:13:44.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.332+0000 7fae05000640 1 -- start start 2024-07-26T06:13:44.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.334+0000 7fae05000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf803bc60 0x7fadf8178f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.334+0000 7fae05000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadf8057830 0x7fadf80e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.334+0000 7fae05000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 0x7fadf80e5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.334+0000 7fae05000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fadf8006bf0 con 0x7fadf80e28f0 2024-07-26T06:13:44.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.334+0000 7fae05000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fadf80e2ee0 con 0x7fadf803bc60 2024-07-26T06:13:44.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.334+0000 7fae05000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fadf8006d70 con 0x7fadf8057830 2024-07-26T06:13:44.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.333+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5abc0066b0 con 0x7f5abc03bc60 2024-07-26T06:13:44.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.335+0000 7fadffe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 0x7fadf80e5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.335+0000 7fadffe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 0x7fadf80e5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45218/0 (socket says 172.21.15.163:45218) 2024-07-26T06:13:44.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.335+0000 7fadffe00640 1 -- 172.21.15.163:0/3179085420 learned_addr learned my addr 172.21.15.163:0/3179085420 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.335+0000 7fadff400640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf803bc60 0x7fadf8178f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.335+0000 7fadfea00640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadf8057830 0x7fadf80e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.337+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5ab4041030 con 0x7f5abc03bc60 2024-07-26T06:13:44.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.337+0000 7fadffe00640 1 -- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadf8057830 msgr2=0x7fadf80e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.337+0000 7fadffe00640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadf8057830 0x7fadf80e52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.337+0000 7fadffe00640 1 -- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf803bc60 msgr2=0x7fadf8178f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.337+0000 7fadffe00640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf803bc60 0x7fadf8178f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.337+0000 7fadffe00640 1 -- 172.21.15.163:0/3179085420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf80e5e10 con 0x7fadf80e28f0 2024-07-26T06:13:44.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.338+0000 7fadffe00640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 0x7fadf80e5800 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fae0006aaf0 tx=0x7fae0006a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.338+0000 7f5aba000640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5aa4076350 0x7f5aa4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.338+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae000690b0 con 0x7fadf80e28f0 2024-07-26T06:13:44.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.338+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf80e60f0 con 0x7fadf80e28f0 2024-07-26T06:13:44.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.339+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5ab407da40 con 0x7f5abc03bc60 2024-07-26T06:13:44.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.339+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf8052770 con 0x7fadf80e28f0 2024-07-26T06:13:44.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.339+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae0008f030 con 0x7fadf80e28f0 2024-07-26T06:13:44.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.339+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00074250 con 0x7fadf80e28f0 2024-07-26T06:13:44.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.339+0000 7f5ac1400640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5aa4076350 0x7f5aa4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.340+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faddc005290 con 0x7fadf80e28f0 2024-07-26T06:13:44.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.343+0000 7f5ac1400640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5aa4076350 0x7f5aa4078810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0007fd0 tx=0x7f5ab0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.342+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fae000743f0 con 0x7fadf80e28f0 2024-07-26T06:13:44.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.343+0000 7fadf3400640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fade00762d0 0x7fade0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.343+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae00111e30 con 0x7fadf80e28f0 2024-07-26T06:13:44.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.344+0000 7fadff400640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fade00762d0 0x7fade0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.345+0000 7fadff400640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fade00762d0 0x7fade0078790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fadf4033000 tx=0x7fadf4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.348+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fae000db0a0 con 0x7fadf80e28f0 2024-07-26T06:13:44.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.350+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ab4081b50 con 0x7f5abc03bc60 2024-07-26T06:13:44.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 -- 172.21.15.163:0/2393817544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff200b53b0 msgr2=0x7eff200b57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 --2- 172.21.15.163:0/2393817544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff200b53b0 0x7eff200b57d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7eff200b6070 tx=0x7eff1002a530 comp rx=0 tx=0).stop 2024-07-26T06:13:44.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 -- 172.21.15.163:0/2393817544 shutdown_connections 2024-07-26T06:13:44.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 --2- 172.21.15.163:0/2393817544 >> v2:172.21.15.163:3300/0 conn(0x7eff2006c270 0x7eff2006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 --2- 172.21.15.163:0/2393817544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff200b53b0 0x7eff200b57d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 --2- 172.21.15.163:0/2393817544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2015d650 0x7eff200d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 -- 172.21.15.163:0/2393817544 >> 172.21.15.163:0/2393817544 conn(0x7eff200d36b0 msgr2=0x7eff200d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 -- 172.21.15.163:0/2393817544 shutdown_connections 2024-07-26T06:13:44.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.383+0000 7eff26000640 1 -- 172.21.15.163:0/2393817544 wait complete. 2024-07-26T06:13:44.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.386+0000 7eff26000640 1 Processor -- start 2024-07-26T06:13:44.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.386+0000 7eff26000640 1 -- start start 2024-07-26T06:13:44.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7eff26000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2006c270 0x7eff2014e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7eff26000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff200b53b0 0x7eff2015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7eff26000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff2015d650 0x7eff200abf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7eff26000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff200d2050 con 0x7eff200b53b0 2024-07-26T06:13:44.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7eff26000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7eff200d1ed0 con 0x7eff2015d650 2024-07-26T06:13:44.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7eff26000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7eff2006cc40 con 0x7eff2006c270 2024-07-26T06:13:44.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.388+0000 7faf02e00640 1 -- 172.21.15.163:0/806379277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b36a0 msgr2=0x7faefc0acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7faf02e00640 1 --2- 172.21.15.163:0/806379277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b36a0 0x7faefc0acec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faef4005fd0 tx=0x7faef402a7c0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7faf02e00640 1 -- 172.21.15.163:0/806379277 shutdown_connections 2024-07-26T06:13:44.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7faf02e00640 1 --2- 172.21.15.163:0/806379277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b36a0 0x7faefc0acec0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7eff1f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2006c270 0x7eff2014e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7faf02e00640 1 --2- 172.21.15.163:0/806379277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faefc0b1b80 0x7faefc15b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7faf02e00640 1 -- 172.21.15.163:0/806379277 >> 172.21.15.163:0/806379277 conn(0x7faefc0b1060 msgr2=0x7faefc0b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7eff1f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2006c270 0x7eff2014e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.163:50922/0 (socket says 172.21.15.163:50922) 2024-07-26T06:13:44.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7eff1f400640 1 -- 172.21.15.163:0/2263855592 learned_addr learned my addr 172.21.15.163:0/2263855592 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7eff1fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff2015d650 0x7eff200abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.389+0000 7eff1ea00640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff200b53b0 0x7eff2015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.390+0000 7faf02e00640 1 -- 172.21.15.163:0/806379277 shutdown_connections 2024-07-26T06:13:44.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.390+0000 7faf02e00640 1 -- 172.21.15.163:0/806379277 wait complete. 2024-07-26T06:13:44.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.390+0000 7eff1fe00640 1 -- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2006c270 msgr2=0x7eff2014e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.390+0000 7eff1fe00640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2006c270 0x7eff2014e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.390+0000 7eff1fe00640 1 -- 172.21.15.163:0/2263855592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff200b53b0 msgr2=0x7eff2015dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.391+0000 7eff1fe00640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff200b53b0 0x7eff2015dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.391+0000 7eff1fe00640 1 -- 172.21.15.163:0/2263855592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2015e620 con 0x7eff2015d650 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.391+0000 7eff1ea00640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff200b53b0 0x7eff2015dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.392+0000 7eff1fe00640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff2015d650 0x7eff200abf20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00e940 tx=0x7eff0c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.392+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00ccc0 con 0x7eff2015d650 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.392+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2015e900 con 0x7eff2015d650 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.392+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff0c004030 con 0x7eff2015d650 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.392+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c004480 con 0x7eff2015d650 2024-07-26T06:13:44.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.392+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2015eec0 con 0x7eff2015d650 2024-07-26T06:13:44.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.394+0000 7faf02e00640 1 Processor -- start 2024-07-26T06:13:44.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.394+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff200b72f0 con 0x7eff2015d650 2024-07-26T06:13:44.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.394+0000 7faf02e00640 1 -- start start 2024-07-26T06:13:44.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.396+0000 7faf02e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b1b80 0x7faefc15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.396+0000 7faf02e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 0x7faefc153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.396+0000 7faf02e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faefc15b380 0x7faefc14c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf02e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faefc0ad580 con 0x7faefc0b36a0 2024-07-26T06:13:44.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf02e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faefc0ad400 con 0x7faefc0b1b80 2024-07-26T06:13:44.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf02e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faefc0ad700 con 0x7faefc15b380 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf01a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b1b80 0x7faefc15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf01000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 0x7faefc153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf01000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 0x7faefc153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45266/0 (socket says 172.21.15.163:45266) 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7faf01000640 1 -- 172.21.15.163:0/1403117779 learned_addr learned my addr 172.21.15.163:0/1403117779 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.398+0000 7faf01000640 1 -- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faefc15b380 msgr2=0x7faefc14c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.398+0000 7faf01000640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faefc15b380 0x7faefc14c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.398+0000 7faf01000640 1 -- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b1b80 msgr2=0x7faefc15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.398+0000 7faf01000640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b1b80 0x7faefc15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.398+0000 7faf01000640 1 -- 172.21.15.163:0/1403117779 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc15ce40 con 0x7faefc0b36a0 2024-07-26T06:13:44.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.397+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff0c0035f0 con 0x7eff2015d650 2024-07-26T06:13:44.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.398+0000 7eff17400640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff00076500 0x7eff000789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7faf01000640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 0x7faefc153d50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faee8014140 tx=0x7faee8014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff0c004220 con 0x7eff2015d650 2024-07-26T06:13:44.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7effa5800640 1 -- 172.21.15.163:0/3642880313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa00b53b0 msgr2=0x7effa00b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7effa5800640 1 --2- 172.21.15.163:0/3642880313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa00b53b0 0x7effa00b5800 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7effa0075ba0 tx=0x7eff8802a630 comp rx=0 tx=0).stop 2024-07-26T06:13:44.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee800fde0 con 0x7faefc0b36a0 2024-07-26T06:13:44.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7eff1f400640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff00076500 0x7eff000789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faee8020070 con 0x7faefc0b36a0 2024-07-26T06:13:44.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.400+0000 7effa5800640 1 -- 172.21.15.163:0/3642880313 shutdown_connections 2024-07-26T06:13:44.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.400+0000 7effa5800640 1 --2- 172.21.15.163:0/3642880313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa006c270 0x7effa006c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.400+0000 7effa5800640 1 --2- 172.21.15.163:0/3642880313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa00b53b0 0x7effa00b5800 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.400+0000 7effa5800640 1 --2- 172.21.15.163:0/3642880313 >> v2:172.21.15.164:3300/0 conn(0x7effa015d650 0x7effa00d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.400+0000 7effa5800640 1 -- 172.21.15.163:0/3642880313 >> 172.21.15.163:0/3642880313 conn(0x7effa00d36b0 msgr2=0x7effa00d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.401+0000 7eff1f400640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff00076500 0x7eff000789c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff18009f50 tx=0x7eff18009680 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8011330 con 0x7faefc0b36a0 2024-07-26T06:13:44.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.399+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faefc15d120 con 0x7faefc0b36a0 2024-07-26T06:13:44.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.400+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faefc15d6e0 con 0x7faefc0b36a0 2024-07-26T06:13:44.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.401+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed8005290 con 0x7faefc0b36a0 2024-07-26T06:13:44.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.402+0000 7effa5800640 1 -- 172.21.15.163:0/3642880313 shutdown_connections 2024-07-26T06:13:44.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.402+0000 7effa5800640 1 -- 172.21.15.163:0/3642880313 wait complete. 2024-07-26T06:13:44.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.405+0000 7effa5800640 1 Processor -- start 2024-07-26T06:13:44.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.403+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faee801a070 con 0x7faefc0b36a0 2024-07-26T06:13:44.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.406+0000 7effa5800640 1 -- start start 2024-07-26T06:13:44.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.405+0000 7faef9600640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faedc0763a0 0x7faedc078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.405+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faee809fac0 con 0x7faefc0b36a0 2024-07-26T06:13:44.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.406+0000 7faf01a00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faedc0763a0 0x7faedc078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.407+0000 7effa5800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa006c270 0x7effa00cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.407+0000 7effa5800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa00b53b0 0x7effa00b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7effa5800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa015d650 0x7effa0072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7effa5800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7effa00d2050 con 0x7effa015d650 2024-07-26T06:13:44.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7effa5800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7effa00d1ed0 con 0x7effa006c270 2024-07-26T06:13:44.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7faf01a00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faedc0763a0 0x7faedc078860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faef0007fd0 tx=0x7faef0001370 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7effa5800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7effa006cc20 con 0x7effa00b53b0 2024-07-26T06:13:44.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7eff9ea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa006c270 0x7effa00cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7eff9f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa015d650 0x7effa0072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7eff9ea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa006c270 0x7effa00cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.163:40906/0 (socket says 172.21.15.163:40906) 2024-07-26T06:13:44.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7eff9ea00640 1 -- 172.21.15.163:0/2862748688 learned_addr learned my addr 172.21.15.163:0/2862748688 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:44.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.408+0000 7eff96000640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa00b53b0 0x7effa00b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff0c060f50 con 0x7eff2015d650 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff9f400640 1 -- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa00b53b0 msgr2=0x7effa00b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff9f400640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa00b53b0 0x7effa00b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff9f400640 1 -- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa006c270 msgr2=0x7effa00cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff9f400640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa006c270 0x7effa00cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff9f400640 1 -- 172.21.15.163:0/2862748688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa00714d0 con 0x7effa015d650 2024-07-26T06:13:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.409+0000 7eff9f400640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa015d650 0x7effa0072570 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff980119b0 tx=0x7eff98015560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.413+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff98004d20 con 0x7effa015d650 2024-07-26T06:13:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.413+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff98005020 con 0x7effa015d650 2024-07-26T06:13:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.413+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff98005450 con 0x7effa015d650 2024-07-26T06:13:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.413+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faee8018040 con 0x7faefc0b36a0 2024-07-26T06:13:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.413+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa00717b0 con 0x7effa015d650 2024-07-26T06:13:44.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.413+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa0071d70 con 0x7effa015d650 2024-07-26T06:13:44.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.417+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff980055f0 con 0x7effa015d650 2024-07-26T06:13:44.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.418+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa00b5800 con 0x7effa015d650 2024-07-26T06:13:44.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.419+0000 7eff97400640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff7c076320 0x7eff7c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:44.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.419+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff9809fd90 con 0x7effa015d650 2024-07-26T06:13:44.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.420+0000 7eff9ea00640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff7c076320 0x7eff7c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:44.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.421+0000 7eff9ea00640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff7c076320 0x7eff7c0787e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff90009c00 tx=0x7eff90005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:44.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.431+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff98018070 con 0x7effa015d650 2024-07-26T06:13:44.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.718+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f16ac005580 con 0x7f16d40b4d40 2024-07-26T06:13:44.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.719+0000 7f16cbe00640 1 -- 172.21.15.163:0/2770012752 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f16cc0613c0 con 0x7f16d40b4d40 2024-07-26T06:13:44.720 INFO:teuthology.orchestra.run.smithi163.stdout:64424509448 2024-07-26T06:13:44.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.722+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f16b40764d0 msgr2=0x7f16b4078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.722+0000 7f16d9400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f16b40764d0 0x7f16b4078990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16c0034000 tx=0x7f16c0004730 comp rx=0 tx=0).stop 2024-07-26T06:13:44.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.723+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d40b4d40 msgr2=0x7f16d40cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.723+0000 7f16d9400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d40b4d40 0x7f16d40cb6c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00c6a0 tx=0x7f16cc00cb60 comp rx=0 tx=0).stop 2024-07-26T06:13:44.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.723+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 shutdown_connections 2024-07-26T06:13:44.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.723+0000 7f16d9400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f16b40764d0 0x7f16b4078990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.723+0000 7f16d9400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16d40b5a80 0x7f16d40cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.724+0000 7f16d9400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16d40b5660 0x7f16d40cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.724+0000 7f16d9400640 1 --2- 172.21.15.163:0/2770012752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16d40b4d40 0x7f16d40cb6c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.724+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 >> 172.21.15.163:0/2770012752 conn(0x7f16d40cfd40 msgr2=0x7f16d4160280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.724+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 shutdown_connections 2024-07-26T06:13:44.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.724+0000 7f16d9400640 1 -- 172.21.15.163:0/2770012752 wait complete. 2024-07-26T06:13:44.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.0 2024-07-26T06:13:44.752 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.771+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f43f00cc010 con 0x7f43f00b2670 2024-07-26T06:13:44.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.772+0000 7f43ecc00640 1 -- 172.21.15.163:0/729302955 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f43e8060560 con 0x7f43f00b2670 2024-07-26T06:13:44.773 INFO:teuthology.orchestra.run.smithi163.stdout:60129542152 2024-07-26T06:13:44.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.774+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7faddc005580 con 0x7fadf80e28f0 2024-07-26T06:13:44.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.774+0000 7fadf3400640 1 -- 172.21.15.163:0/3179085420 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fae000df3a0 con 0x7fadf80e28f0 2024-07-26T06:13:44.775 INFO:teuthology.orchestra.run.smithi163.stdout:64424509449 2024-07-26T06:13:44.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43d0076500 msgr2=0x7f43d00789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43d0076500 0x7f43d00789c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f43f0168a60 tx=0x7f43e000b2e0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 msgr2=0x7f43f0209130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f0209130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7a0 tx=0x7f43e800cc60 comp rx=0 tx=0).stop 2024-07-26T06:13:44.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 shutdown_connections 2024-07-26T06:13:44.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43d0076500 0x7f43d00789c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43f016c410 0x7f43f01696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43f00cba90 0x7f43f0168da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 --2- 172.21.15.163:0/729302955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00b2670 0x7f43f0209130 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7a0 tx=0x7f43e800cc60 comp rx=0 tx=0).stop 2024-07-26T06:13:44.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.776+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 >> 172.21.15.163:0/729302955 conn(0x7f43f006d280 msgr2=0x7f43f00af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.777+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 shutdown_connections 2024-07-26T06:13:44.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.777+0000 7f43f6200640 1 -- 172.21.15.163:0/729302955 wait complete. 2024-07-26T06:13:44.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.777+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fade00762d0 msgr2=0x7fade0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.777+0000 7fae05000640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fade00762d0 0x7fade0078790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fadf4033000 tx=0x7fadf4004730 comp rx=0 tx=0).stop 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.777+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 msgr2=0x7fadf80e5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.777+0000 7fae05000640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 0x7fadf80e5800 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fae0006aaf0 tx=0x7fae0006a5c0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 shutdown_connections 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fade00762d0 0x7fade0078790 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fadf4033000 tx=0x7fadf4004730 comp rx=0 tx=0).stop 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80e28f0 0x7fadf80e5800 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadf8057830 0x7fadf80e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 --2- 172.21.15.163:0/3179085420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadf803bc60 0x7fadf8178f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 >> 172.21.15.163:0/3179085420 conn(0x7fadf803a520 msgr2=0x7fadf8004f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 shutdown_connections 2024-07-26T06:13:44.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.778+0000 7fae05000640 1 -- 172.21.15.163:0/3179085420 wait complete. 2024-07-26T06:13:44.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.2 2024-07-26T06:13:44.796 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-07-26T06:13:44.802 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.806+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5abc03e9b0 con 0x7f5abc03bc60 2024-07-26T06:13:44.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.807+0000 7f5aba000640 1 -- 172.21.15.163:0/711480920 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5ab4081450 con 0x7f5abc03bc60 2024-07-26T06:13:44.808 INFO:teuthology.orchestra.run.smithi163.stdout:60129542150 2024-07-26T06:13:44.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5aa4076350 msgr2=0x7f5aa4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5aa4076350 0x7f5aa4078810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0007fd0 tx=0x7f5ab0009040 comp rx=0 tx=0).stop 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 msgr2=0x7f5abc0e3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 0x7f5abc0e3060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4034040 tx=0x7f5ab4003910 comp rx=0 tx=0).stop 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 shutdown_connections 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5aa4076350 0x7f5aa4078810 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5abc0e28f0 0x7f5abc0e4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5abc057830 0x7f5abc0d2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 --2- 172.21.15.163:0/711480920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5abc03bc60 0x7f5abc0e3060 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 >> 172.21.15.163:0/711480920 conn(0x7f5abc03a520 msgr2=0x7f5abc039d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.810+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 shutdown_connections 2024-07-26T06:13:44.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.811+0000 7f5ac3200640 1 -- 172.21.15.163:0/711480920 wait complete. 2024-07-26T06:13:44.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.819+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb5d4005580 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.819+0000 7fb5f3e00640 1 -- 172.21.15.163:0/282580669 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb5e406d8a0 con 0x7fb5fc0b1e80 2024-07-26T06:13:44.820 INFO:teuthology.orchestra.run.smithi163.stdout:60129542151 2024-07-26T06:13:44.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.822+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb5d80764d0 msgr2=0x7fb5d8078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.822+0000 7fb600a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb5d80764d0 0x7fb5d8078990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec034000 tx=0x7fb5ec004730 comp rx=0 tx=0).stop 2024-07-26T06:13:44.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.822+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 msgr2=0x7fb5fc156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.822+0000 7fb600a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 0x7fb5fc156ce0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400e7c0 tx=0x7fb5e400ec80 comp rx=0 tx=0).stop 2024-07-26T06:13:44.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 shutdown_connections 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb5d80764d0 0x7fb5d8078990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5fc15d5a0 0x7fb5fc0c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5fc0ce830 0x7fb5fc0c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 --2- 172.21.15.163:0/282580669 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5fc0b1e80 0x7fb5fc156ce0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 >> 172.21.15.163:0/282580669 conn(0x7fb5fc0cfda0 msgr2=0x7fb5fc15aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 shutdown_connections 2024-07-26T06:13:44.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.823+0000 7fb600a00640 1 -- 172.21.15.163:0/282580669 wait complete. 2024-07-26T06:13:44.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.832+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7faed8005580 con 0x7faefc0b36a0 2024-07-26T06:13:44.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.832+0000 7faef9600640 1 -- 172.21.15.163:0/1403117779 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faee8069490 con 0x7faefc0b36a0 2024-07-26T06:13:44.833 INFO:teuthology.orchestra.run.smithi163.stdout:64424509448 2024-07-26T06:13:44.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faedc0763a0 msgr2=0x7faedc078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faedc0763a0 0x7faedc078860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faef0007fd0 tx=0x7faef0001370 comp rx=0 tx=0).stop 2024-07-26T06:13:44.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 msgr2=0x7faefc153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 0x7faefc153d50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faee8014140 tx=0x7faee8014600 comp rx=0 tx=0).stop 2024-07-26T06:13:44.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 shutdown_connections 2024-07-26T06:13:44.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faedc0763a0 0x7faedc078860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faefc15b380 0x7faefc14c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faefc0b36a0 0x7faefc153d50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 --2- 172.21.15.163:0/1403117779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faefc0b1b80 0x7faefc15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 >> 172.21.15.163:0/1403117779 conn(0x7faefc0b1060 msgr2=0x7faefc0aefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.3 2024-07-26T06:13:44.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 shutdown_connections 2024-07-26T06:13:44.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.836+0000 7faf02e00640 1 -- 172.21.15.163:0/1403117779 wait complete. 2024-07-26T06:13:44.839 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.846+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7effa0159db0 con 0x7effa015d650 2024-07-26T06:13:44.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2024-07-26T06:13:44.848 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.848+0000 7eff97400640 1 -- 172.21.15.163:0/2862748688 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff980696d0 con 0x7effa015d650 2024-07-26T06:13:44.849 INFO:teuthology.orchestra.run.smithi163.stdout:64424509447 2024-07-26T06:13:44.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7eff2006c760 con 0x7eff2015d650 2024-07-26T06:13:44.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff7c076320 msgr2=0x7eff7c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff7c076320 0x7eff7c0787e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff90009c00 tx=0x7eff90005f90 comp rx=0 tx=0).stop 2024-07-26T06:13:44.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa015d650 msgr2=0x7effa0072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa015d650 0x7effa0072570 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff980119b0 tx=0x7eff98015560 comp rx=0 tx=0).stop 2024-07-26T06:13:44.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 shutdown_connections 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff7c076320 0x7eff7c0787e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effa015d650 0x7effa0072570 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa00b53b0 0x7effa00b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 --2- 172.21.15.163:0/2862748688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa006c270 0x7effa00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 >> 172.21.15.163:0/2862748688 conn(0x7effa00d36b0 msgr2=0x7effa00d3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.850+0000 7eff17400640 1 -- 172.21.15.163:0/2263855592 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff0c065250 con 0x7eff2015d650 2024-07-26T06:13:44.852 INFO:teuthology.orchestra.run.smithi163.stdout:64424509447 2024-07-26T06:13:44.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.851+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 shutdown_connections 2024-07-26T06:13:44.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.851+0000 7effa5800640 1 -- 172.21.15.163:0/2862748688 wait complete. 2024-07-26T06:13:44.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff00076500 msgr2=0x7eff000789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff00076500 0x7eff000789c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff18009f50 tx=0x7eff18009680 comp rx=0 tx=0).stop 2024-07-26T06:13:44.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff2015d650 msgr2=0x7eff200abf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:44.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff2015d650 0x7eff200abf20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00e940 tx=0x7eff0c00ee00 comp rx=0 tx=0).stop 2024-07-26T06:13:44.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 shutdown_connections 2024-07-26T06:13:44.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff00076500 0x7eff000789c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff2015d650 0x7eff200abf20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff200b53b0 0x7eff2015dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 --2- 172.21.15.163:0/2263855592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff2006c270 0x7eff2014e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:44.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 >> 172.21.15.163:0/2263855592 conn(0x7eff200d36b0 msgr2=0x7eff200b78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:44.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 shutdown_connections 2024-07-26T06:13:44.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:44.853+0000 7eff26000640 1 -- 172.21.15.163:0/2263855592 wait complete. 2024-07-26T06:13:44.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.6 2024-07-26T06:13:44.856 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.7 2024-07-26T06:13:44.869 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.870 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.5 2024-07-26T06:13:44.870 DEBUG:teuthology.parallel:result is None 2024-07-26T06:13:44.870 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:13:44.871 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:13:45.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 -- 172.21.15.163:0/1911265974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040b1e80 msgr2=0x7f49040b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 --2- 172.21.15.163:0/1911265974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040b1e80 0x7f49040b6a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f49040690b0 tx=0x7f48fc02a440 comp rx=0 tx=0).stop 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 -- 172.21.15.163:0/1911265974 shutdown_connections 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 --2- 172.21.15.163:0/1911265974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040b1e80 0x7f49040b6a00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 --2- 172.21.15.163:0/1911265974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49040ce830 0x7f49040b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 --2- 172.21.15.163:0/1911265974 >> v2:172.21.15.164:3300/0 conn(0x7f490415d5a0 0x7f49040ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 -- 172.21.15.163:0/1911265974 >> 172.21.15.163:0/1911265974 conn(0x7f49040cfda0 msgr2=0x7f49040d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 -- 172.21.15.163:0/1911265974 shutdown_connections 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.098+0000 7f490a800640 1 -- 172.21.15.163:0/1911265974 wait complete. 2024-07-26T06:13:45.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.100+0000 7f490a800640 1 Processor -- start 2024-07-26T06:13:45.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.100+0000 7f490a800640 1 -- start start 2024-07-26T06:13:45.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f490a800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49040b1e80 0x7f49040c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f490a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 0x7f490415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f490a800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f490415d5a0 0x7f490415df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f490a800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49040b4da0 con 0x7f49040ce830 2024-07-26T06:13:45.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f490a800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f49040b2630 con 0x7f49040b1e80 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f490a800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f49040b4f20 con 0x7f490415d5a0 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 0x7f490415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 0x7f490415d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36280/0 (socket says 172.21.15.163:36280) 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 -- 172.21.15.163:0/4184089111 learned_addr learned my addr 172.21.15.163:0/4184089111 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4909400640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49040b1e80 0x7f49040c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4909e00640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f490415d5a0 0x7f490415df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 -- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f490415d5a0 msgr2=0x7f490415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f490415d5a0 0x7f490415df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 -- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49040b1e80 msgr2=0x7f49040c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49040b1e80 0x7f49040c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4908a00640 1 -- 172.21.15.163:0/4184089111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49041f3ea0 con 0x7f49040ce830 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.101+0000 7f4909400640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49040b1e80 0x7f49040c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:45.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.102+0000 7f4908a00640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 0x7f490415d9d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f48f000cce0 tx=0x7f48f0007610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:45.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.102+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0003990 con 0x7f49040ce830 2024-07-26T06:13:45.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.102+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48f0003c20 con 0x7f49040ce830 2024-07-26T06:13:45.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.102+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f0004020 con 0x7f49040ce830 2024-07-26T06:13:45.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.102+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49040c7e60 con 0x7f49040ce830 2024-07-26T06:13:45.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.102+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49040c8420 con 0x7f49040ce830 2024-07-26T06:13:45.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.103+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49040ced00 con 0x7f49040ce830 2024-07-26T06:13:45.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.104+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f48f00041c0 con 0x7f49040ce830 2024-07-26T06:13:45.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.105+0000 7f4901600640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f48e40762d0 0x7f48e4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.105+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f48f0097f90 con 0x7f49040ce830 2024-07-26T06:13:45.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.105+0000 7f4909400640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f48e40762d0 0x7f48e4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.106+0000 7f4909400640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f48e40762d0 0x7f48e4078790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f48f40058c0 tx=0x7f48f4005c90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:45.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.109+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f49040ced00 con 0x7f49040ce830 2024-07-26T06:13:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.298+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49040d1230 con 0x7f48e40762d0 2024-07-26T06:13:45.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.311+0000 7f4901600640 1 -- 172.21.15.163:0/4184089111 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f49040d1230 con 0x7f48e40762d0 2024-07-26T06:13:45.314 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:45.317 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f48e40762d0 msgr2=0x7f48e4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f48e40762d0 0x7f48e4078790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f48f40058c0 tx=0x7f48f4005c90 comp rx=0 tx=0).stop 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 msgr2=0x7f490415d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 0x7f490415d9d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f48f000cce0 tx=0x7f48f0007610 comp rx=0 tx=0).stop 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 shutdown_connections 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f48e40762d0 0x7f48e4078790 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f490415d5a0 0x7f490415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49040ce830 0x7f490415d9d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 --2- 172.21.15.163:0/4184089111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49040b1e80 0x7f49040c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 >> 172.21.15.163:0/4184089111 conn(0x7f49040cfda0 msgr2=0x7f49040d1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:45.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 shutdown_connections 2024-07-26T06:13:45.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.318+0000 7f490a800640 1 -- 172.21.15.163:0/4184089111 wait complete. 2024-07-26T06:13:45.335 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-07-26T06:13:43.646969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219036,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755684,"statfs":{"total":773094113280,"available":772869820416,"internally_reserved":0,"allocated":2924544,"data_stored":1765884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.621179"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267640+0000","last_change":"2024-07-26T06:13:35.072083+0000","last_active":"2024-07-26T06:13:37.267640+0000","last_peered":"2024-07-26T06:13:37.267640+0000","last_clean":"2024-07-26T06:13:37.267640+0000","last_became_active":"2024-07-26T06:13:35.071746+0000","last_became_peered":"2024-07-26T06:13:35.071746+0000","last_unstale":"2024-07-26T06:13:37.267640+0000","last_undegraded":"2024-07-26T06:13:37.267640+0000","last_fullsized":"2024-07-26T06:13:37.267640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046586+0000","last_change":"2024-07-26T06:13:35.058738+0000","last_active":"2024-07-26T06:13:37.046586+0000","last_peered":"2024-07-26T06:13:37.046586+0000","last_clean":"2024-07-26T06:13:37.046586+0000","last_became_active":"2024-07-26T06:13:35.058470+0000","last_became_peered":"2024-07-26T06:13:35.058470+0000","last_unstale":"2024-07-26T06:13:37.046586+0000","last_undegraded":"2024-07-26T06:13:37.046586+0000","last_fullsized":"2024-07-26T06:13:37.046586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:49.213504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056891+0000","last_change":"2024-07-26T06:13:35.042867+0000","last_active":"2024-07-26T06:13:36.056891+0000","last_peered":"2024-07-26T06:13:36.056891+0000","last_clean":"2024-07-26T06:13:36.056891+0000","last_became_active":"2024-07-26T06:13:35.042660+0000","last_became_peered":"2024-07-26T06:13:35.042660+0000","last_unstale":"2024-07-26T06:13:36.056891+0000","last_undegraded":"2024-07-26T06:13:36.056891+0000","last_fullsized":"2024-07-26T06:13:36.056891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:17:02.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165050+0000","last_change":"2024-07-26T06:13:35.076112+0000","last_active":"2024-07-26T06:13:36.165050+0000","last_peered":"2024-07-26T06:13:36.165050+0000","last_clean":"2024-07-26T06:13:36.165050+0000","last_became_active":"2024-07-26T06:13:35.075903+0000","last_became_peered":"2024-07-26T06:13:35.075903+0000","last_unstale":"2024-07-26T06:13:36.165050+0000","last_undegraded":"2024-07-26T06:13:36.165050+0000","last_fullsized":"2024-07-26T06:13:36.165050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267271+0000","last_change":"2024-07-26T06:13:35.068273+0000","last_active":"2024-07-26T06:13:37.267271+0000","last_peered":"2024-07-26T06:13:37.267271+0000","last_clean":"2024-07-26T06:13:37.267271+0000","last_became_active":"2024-07-26T06:13:35.067645+0000","last_became_peered":"2024-07-26T06:13:35.067645+0000","last_unstale":"2024-07-26T06:13:37.267271+0000","last_undegraded":"2024-07-26T06:13:37.267271+0000","last_fullsized":"2024-07-26T06:13:37.267271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164727+0000","last_change":"2024-07-26T06:13:35.080356+0000","last_active":"2024-07-26T06:13:36.164727+0000","last_peered":"2024-07-26T06:13:36.164727+0000","last_clean":"2024-07-26T06:13:36.164727+0000","last_became_active":"2024-07-26T06:13:35.079906+0000","last_became_peered":"2024-07-26T06:13:35.079906+0000","last_unstale":"2024-07-26T06:13:36.164727+0000","last_undegraded":"2024-07-26T06:13:36.164727+0000","last_fullsized":"2024-07-26T06:13:36.164727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266764+0000","last_change":"2024-07-26T06:13:35.070227+0000","last_active":"2024-07-26T06:13:37.266764+0000","last_peered":"2024-07-26T06:13:37.266764+0000","last_clean":"2024-07-26T06:13:37.266764+0000","last_became_active":"2024-07-26T06:13:35.069602+0000","last_became_peered":"2024-07-26T06:13:35.069602+0000","last_unstale":"2024-07-26T06:13:37.266764+0000","last_undegraded":"2024-07-26T06:13:37.266764+0000","last_fullsized":"2024-07-26T06:13:37.266764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059249+0000","last_change":"2024-07-26T06:13:35.077098+0000","last_active":"2024-07-26T06:13:36.059249+0000","last_peered":"2024-07-26T06:13:36.059249+0000","last_clean":"2024-07-26T06:13:36.059249+0000","last_became_active":"2024-07-26T06:13:35.076814+0000","last_became_peered":"2024-07-26T06:13:35.076814+0000","last_unstale":"2024-07-26T06:13:36.059249+0000","last_undegraded":"2024-07-26T06:13:36.059249+0000","last_fullsized":"2024-07-26T06:13:36.059249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:19.325249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267856+0000","last_change":"2024-07-26T06:13:35.081415+0000","last_active":"2024-07-26T06:13:37.267856+0000","last_peered":"2024-07-26T06:13:37.267856+0000","last_clean":"2024-07-26T06:13:37.267856+0000","last_became_active":"2024-07-26T06:13:35.080563+0000","last_became_peered":"2024-07-26T06:13:35.080563+0000","last_unstale":"2024-07-26T06:13:37.267856+0000","last_undegraded":"2024-07-26T06:13:37.267856+0000","last_fullsized":"2024-07-26T06:13:37.267856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241645+0000","last_change":"2024-07-26T06:13:35.067607+0000","last_active":"2024-07-26T06:13:36.241645+0000","last_peered":"2024-07-26T06:13:36.241645+0000","last_clean":"2024-07-26T06:13:36.241645+0000","last_became_active":"2024-07-26T06:13:35.064672+0000","last_became_peered":"2024-07-26T06:13:35.064672+0000","last_unstale":"2024-07-26T06:13:36.241645+0000","last_undegraded":"2024-07-26T06:13:36.241645+0000","last_fullsized":"2024-07-26T06:13:36.241645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241131+0000","last_change":"2024-07-26T06:13:35.070018+0000","last_active":"2024-07-26T06:13:36.241131+0000","last_peered":"2024-07-26T06:13:36.241131+0000","last_clean":"2024-07-26T06:13:36.241131+0000","last_became_active":"2024-07-26T06:13:35.069721+0000","last_became_peered":"2024-07-26T06:13:35.069721+0000","last_unstale":"2024-07-26T06:13:36.241131+0000","last_undegraded":"2024-07-26T06:13:36.241131+0000","last_fullsized":"2024-07-26T06:13:36.241131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063011+0000","last_change":"2024-07-26T06:13:35.062741+0000","last_active":"2024-07-26T06:13:36.063011+0000","last_peered":"2024-07-26T06:13:36.063011+0000","last_clean":"2024-07-26T06:13:36.063011+0000","last_became_active":"2024-07-26T06:13:35.061208+0000","last_became_peered":"2024-07-26T06:13:35.061208+0000","last_unstale":"2024-07-26T06:13:36.063011+0000","last_undegraded":"2024-07-26T06:13:36.063011+0000","last_fullsized":"2024-07-26T06:13:36.063011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:50:23.338780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059514+0000","last_change":"2024-07-26T06:13:35.061997+0000","last_active":"2024-07-26T06:13:36.059514+0000","last_peered":"2024-07-26T06:13:36.059514+0000","last_clean":"2024-07-26T06:13:36.059514+0000","last_became_active":"2024-07-26T06:13:35.061691+0000","last_became_peered":"2024-07-26T06:13:35.061691+0000","last_unstale":"2024-07-26T06:13:36.059514+0000","last_undegraded":"2024-07-26T06:13:36.059514+0000","last_fullsized":"2024-07-26T06:13:36.059514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:00.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056756+0000","last_change":"2024-07-26T06:13:35.060288+0000","last_active":"2024-07-26T06:13:36.056756+0000","last_peered":"2024-07-26T06:13:36.056756+0000","last_clean":"2024-07-26T06:13:36.056756+0000","last_became_active":"2024-07-26T06:13:35.059968+0000","last_became_peered":"2024-07-26T06:13:35.059968+0000","last_unstale":"2024-07-26T06:13:36.056756+0000","last_undegraded":"2024-07-26T06:13:36.056756+0000","last_fullsized":"2024-07-26T06:13:36.056756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:23:13.174113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045688+0000","last_change":"2024-07-26T06:13:35.063974+0000","last_active":"2024-07-26T06:13:37.045688+0000","last_peered":"2024-07-26T06:13:37.045688+0000","last_clean":"2024-07-26T06:13:37.045688+0000","last_became_active":"2024-07-26T06:13:35.062227+0000","last_became_peered":"2024-07-26T06:13:35.062227+0000","last_unstale":"2024-07-26T06:13:37.045688+0000","last_undegraded":"2024-07-26T06:13:37.045688+0000","last_fullsized":"2024-07-26T06:13:37.045688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:19.325249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165749+0000","last_change":"2024-07-26T06:13:35.079657+0000","last_active":"2024-07-26T06:13:36.165749+0000","last_peered":"2024-07-26T06:13:36.165749+0000","last_clean":"2024-07-26T06:13:36.165749+0000","last_became_active":"2024-07-26T06:13:35.078980+0000","last_became_peered":"2024-07-26T06:13:35.078980+0000","last_unstale":"2024-07-26T06:13:36.165749+0000","last_undegraded":"2024-07-26T06:13:36.165749+0000","last_fullsized":"2024-07-26T06:13:36.165749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165976+0000","last_change":"2024-07-26T06:13:35.081733+0000","last_active":"2024-07-26T06:13:36.165976+0000","last_peered":"2024-07-26T06:13:36.165976+0000","last_clean":"2024-07-26T06:13:36.165976+0000","last_became_active":"2024-07-26T06:13:35.081316+0000","last_became_peered":"2024-07-26T06:13:35.081316+0000","last_unstale":"2024-07-26T06:13:36.165976+0000","last_undegraded":"2024-07-26T06:13:36.165976+0000","last_fullsized":"2024-07-26T06:13:36.165976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266057+0000","last_change":"2024-07-26T06:13:35.062888+0000","last_active":"2024-07-26T06:13:37.266057+0000","last_peered":"2024-07-26T06:13:37.266057+0000","last_clean":"2024-07-26T06:13:37.266057+0000","last_became_active":"2024-07-26T06:13:35.062617+0000","last_became_peered":"2024-07-26T06:13:35.062617+0000","last_unstale":"2024-07-26T06:13:37.266057+0000","last_undegraded":"2024-07-26T06:13:37.266057+0000","last_fullsized":"2024-07-26T06:13:37.266057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:28.831320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268031+0000","last_change":"2024-07-26T06:13:35.073326+0000","last_active":"2024-07-26T06:13:37.268031+0000","last_peered":"2024-07-26T06:13:37.268031+0000","last_clean":"2024-07-26T06:13:37.268031+0000","last_became_active":"2024-07-26T06:13:35.073019+0000","last_became_peered":"2024-07-26T06:13:35.073019+0000","last_unstale":"2024-07-26T06:13:37.268031+0000","last_undegraded":"2024-07-26T06:13:37.268031+0000","last_fullsized":"2024-07-26T06:13:37.268031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241158+0000","last_change":"2024-07-26T06:13:35.067632+0000","last_active":"2024-07-26T06:13:36.241158+0000","last_peered":"2024-07-26T06:13:36.241158+0000","last_clean":"2024-07-26T06:13:36.241158+0000","last_became_active":"2024-07-26T06:13:35.066527+0000","last_became_peered":"2024-07-26T06:13:35.066527+0000","last_unstale":"2024-07-26T06:13:36.241158+0000","last_undegraded":"2024-07-26T06:13:36.241158+0000","last_fullsized":"2024-07-26T06:13:36.241158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057001+0000","last_change":"2024-07-26T06:13:35.056477+0000","last_active":"2024-07-26T06:13:36.057001+0000","last_peered":"2024-07-26T06:13:36.057001+0000","last_clean":"2024-07-26T06:13:36.057001+0000","last_became_active":"2024-07-26T06:13:35.056222+0000","last_became_peered":"2024-07-26T06:13:35.056222+0000","last_unstale":"2024-07-26T06:13:36.057001+0000","last_undegraded":"2024-07-26T06:13:36.057001+0000","last_fullsized":"2024-07-26T06:13:36.057001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:20.776784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164819+0000","last_change":"2024-07-26T06:13:35.081680+0000","last_active":"2024-07-26T06:13:36.164819+0000","last_peered":"2024-07-26T06:13:36.164819+0000","last_clean":"2024-07-26T06:13:36.164819+0000","last_became_active":"2024-07-26T06:13:35.081409+0000","last_became_peered":"2024-07-26T06:13:35.081409+0000","last_unstale":"2024-07-26T06:13:36.164819+0000","last_undegraded":"2024-07-26T06:13:36.164819+0000","last_fullsized":"2024-07-26T06:13:36.164819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059747+0000","last_change":"2024-07-26T06:13:35.063598+0000","last_active":"2024-07-26T06:13:36.059747+0000","last_peered":"2024-07-26T06:13:36.059747+0000","last_clean":"2024-07-26T06:13:36.059747+0000","last_became_active":"2024-07-26T06:13:35.063108+0000","last_became_peered":"2024-07-26T06:13:35.063108+0000","last_unstale":"2024-07-26T06:13:36.059747+0000","last_undegraded":"2024-07-26T06:13:36.059747+0000","last_fullsized":"2024-07-26T06:13:36.059747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:49.213504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164852+0000","last_change":"2024-07-26T06:13:35.078539+0000","last_active":"2024-07-26T06:13:36.164852+0000","last_peered":"2024-07-26T06:13:36.164852+0000","last_clean":"2024-07-26T06:13:36.164852+0000","last_became_active":"2024-07-26T06:13:35.074963+0000","last_became_peered":"2024-07-26T06:13:35.074963+0000","last_unstale":"2024-07-26T06:13:36.164852+0000","last_undegraded":"2024-07-26T06:13:36.164852+0000","last_fullsized":"2024-07-26T06:13:36.164852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060557+0000","last_change":"2024-07-26T06:13:35.053573+0000","last_active":"2024-07-26T06:13:36.060557+0000","last_peered":"2024-07-26T06:13:36.060557+0000","last_clean":"2024-07-26T06:13:36.060557+0000","last_became_active":"2024-07-26T06:13:35.053330+0000","last_became_peered":"2024-07-26T06:13:35.053330+0000","last_unstale":"2024-07-26T06:13:36.060557+0000","last_undegraded":"2024-07-26T06:13:36.060557+0000","last_fullsized":"2024-07-26T06:13:36.060557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:01.836541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241629+0000","last_change":"2024-07-26T06:13:35.074100+0000","last_active":"2024-07-26T06:13:36.241629+0000","last_peered":"2024-07-26T06:13:36.241629+0000","last_clean":"2024-07-26T06:13:36.241629+0000","last_became_active":"2024-07-26T06:13:35.073532+0000","last_became_peered":"2024-07-26T06:13:35.073532+0000","last_unstale":"2024-07-26T06:13:36.241629+0000","last_undegraded":"2024-07-26T06:13:36.241629+0000","last_fullsized":"2024-07-26T06:13:36.241629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046103+0000","last_change":"2024-07-26T06:13:35.053991+0000","last_active":"2024-07-26T06:13:37.046103+0000","last_peered":"2024-07-26T06:13:37.046103+0000","last_clean":"2024-07-26T06:13:37.046103+0000","last_became_active":"2024-07-26T06:13:35.053723+0000","last_became_peered":"2024-07-26T06:13:35.053723+0000","last_unstale":"2024-07-26T06:13:37.046103+0000","last_undegraded":"2024-07-26T06:13:37.046103+0000","last_fullsized":"2024-07-26T06:13:37.046103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:27.327738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060277+0000","last_change":"2024-07-26T06:13:35.055687+0000","last_active":"2024-07-26T06:13:36.060277+0000","last_peered":"2024-07-26T06:13:36.060277+0000","last_clean":"2024-07-26T06:13:36.060277+0000","last_became_active":"2024-07-26T06:13:35.054863+0000","last_became_peered":"2024-07-26T06:13:35.054863+0000","last_unstale":"2024-07-26T06:13:36.060277+0000","last_undegraded":"2024-07-26T06:13:36.060277+0000","last_fullsized":"2024-07-26T06:13:36.060277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.827844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059098+0000","last_change":"2024-07-26T06:13:35.047582+0000","last_active":"2024-07-26T06:13:36.059098+0000","last_peered":"2024-07-26T06:13:36.059098+0000","last_clean":"2024-07-26T06:13:36.059098+0000","last_became_active":"2024-07-26T06:13:35.047322+0000","last_became_peered":"2024-07-26T06:13:35.047322+0000","last_unstale":"2024-07-26T06:13:36.059098+0000","last_undegraded":"2024-07-26T06:13:36.059098+0000","last_fullsized":"2024-07-26T06:13:36.059098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165246+0000","last_change":"2024-07-26T06:13:35.079808+0000","last_active":"2024-07-26T06:13:36.165246+0000","last_peered":"2024-07-26T06:13:36.165246+0000","last_clean":"2024-07-26T06:13:36.165246+0000","last_became_active":"2024-07-26T06:13:35.076583+0000","last_became_peered":"2024-07-26T06:13:35.076583+0000","last_unstale":"2024-07-26T06:13:36.165246+0000","last_undegraded":"2024-07-26T06:13:36.165246+0000","last_fullsized":"2024-07-26T06:13:36.165246+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265681+0000","last_change":"2024-07-26T06:13:35.062887+0000","last_active":"2024-07-26T06:13:37.265681+0000","last_peered":"2024-07-26T06:13:37.265681+0000","last_clean":"2024-07-26T06:13:37.265681+0000","last_became_active":"2024-07-26T06:13:35.062553+0000","last_became_peered":"2024-07-26T06:13:35.062553+0000","last_unstale":"2024-07-26T06:13:37.265681+0000","last_undegraded":"2024-07-26T06:13:37.265681+0000","last_fullsized":"2024-07-26T06:13:37.265681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:38.675161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240837+0000","last_change":"2024-07-26T06:13:35.059916+0000","last_active":"2024-07-26T06:13:36.240837+0000","last_peered":"2024-07-26T06:13:36.240837+0000","last_clean":"2024-07-26T06:13:36.240837+0000","last_became_active":"2024-07-26T06:13:35.058276+0000","last_became_peered":"2024-07-26T06:13:35.058276+0000","last_unstale":"2024-07-26T06:13:36.240837+0000","last_undegraded":"2024-07-26T06:13:36.240837+0000","last_fullsized":"2024-07-26T06:13:36.240837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062155+0000","last_change":"2024-07-26T06:13:35.052170+0000","last_active":"2024-07-26T06:13:36.062155+0000","last_peered":"2024-07-26T06:13:36.062155+0000","last_clean":"2024-07-26T06:13:36.062155+0000","last_became_active":"2024-07-26T06:13:35.051630+0000","last_became_peered":"2024-07-26T06:13:35.051630+0000","last_unstale":"2024-07-26T06:13:36.062155+0000","last_undegraded":"2024-07-26T06:13:36.062155+0000","last_fullsized":"2024-07-26T06:13:36.062155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:31.103018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059899+0000","last_change":"2024-07-26T06:13:34.065590+0000","last_active":"2024-07-26T06:13:36.059899+0000","last_peered":"2024-07-26T06:13:36.059899+0000","last_clean":"2024-07-26T06:13:36.059899+0000","last_became_active":"2024-07-26T06:13:34.065275+0000","last_became_peered":"2024-07-26T06:13:34.065275+0000","last_unstale":"2024-07-26T06:13:36.059899+0000","last_undegraded":"2024-07-26T06:13:36.059899+0000","last_fullsized":"2024-07-26T06:13:36.059899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059022+0000","last_change":"2024-07-26T06:13:35.057161+0000","last_active":"2024-07-26T06:13:36.059022+0000","last_peered":"2024-07-26T06:13:36.059022+0000","last_clean":"2024-07-26T06:13:36.059022+0000","last_became_active":"2024-07-26T06:13:35.056843+0000","last_became_peered":"2024-07-26T06:13:35.056843+0000","last_unstale":"2024-07-26T06:13:36.059022+0000","last_undegraded":"2024-07-26T06:13:36.059022+0000","last_fullsized":"2024-07-26T06:13:36.059022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241143+0000","last_change":"2024-07-26T06:13:34.061026+0000","last_active":"2024-07-26T06:13:36.241143+0000","last_peered":"2024-07-26T06:13:36.241143+0000","last_clean":"2024-07-26T06:13:36.241143+0000","last_became_active":"2024-07-26T06:13:34.060729+0000","last_became_peered":"2024-07-26T06:13:34.060729+0000","last_unstale":"2024-07-26T06:13:36.241143+0000","last_undegraded":"2024-07-26T06:13:36.241143+0000","last_fullsized":"2024-07-26T06:13:36.241143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059737+0000","last_change":"2024-07-26T06:13:34.053732+0000","last_active":"2024-07-26T06:13:36.059737+0000","last_peered":"2024-07-26T06:13:36.059737+0000","last_clean":"2024-07-26T06:13:36.059737+0000","last_became_active":"2024-07-26T06:13:34.053385+0000","last_became_peered":"2024-07-26T06:13:34.053385+0000","last_unstale":"2024-07-26T06:13:36.059737+0000","last_undegraded":"2024-07-26T06:13:36.059737+0000","last_fullsized":"2024-07-26T06:13:36.059737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:24:03.458270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059243+0000","last_change":"2024-07-26T06:13:35.042064+0000","last_active":"2024-07-26T06:13:36.059243+0000","last_peered":"2024-07-26T06:13:36.059243+0000","last_clean":"2024-07-26T06:13:36.059243+0000","last_became_active":"2024-07-26T06:13:35.041741+0000","last_became_peered":"2024-07-26T06:13:35.041741+0000","last_unstale":"2024-07-26T06:13:36.059243+0000","last_undegraded":"2024-07-26T06:13:36.059243+0000","last_fullsized":"2024-07-26T06:13:36.059243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:20.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165112+0000","last_change":"2024-07-26T06:13:34.061502+0000","last_active":"2024-07-26T06:13:36.165112+0000","last_peered":"2024-07-26T06:13:36.165112+0000","last_clean":"2024-07-26T06:13:36.165112+0000","last_became_active":"2024-07-26T06:13:34.061155+0000","last_became_peered":"2024-07-26T06:13:34.061155+0000","last_unstale":"2024-07-26T06:13:36.165112+0000","last_undegraded":"2024-07-26T06:13:36.165112+0000","last_fullsized":"2024-07-26T06:13:36.165112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045692+0000","last_change":"2024-07-26T06:13:35.051729+0000","last_active":"2024-07-26T06:13:37.045692+0000","last_peered":"2024-07-26T06:13:37.045692+0000","last_clean":"2024-07-26T06:13:37.045692+0000","last_became_active":"2024-07-26T06:13:35.051429+0000","last_became_peered":"2024-07-26T06:13:35.051429+0000","last_unstale":"2024-07-26T06:13:37.045692+0000","last_undegraded":"2024-07-26T06:13:37.045692+0000","last_fullsized":"2024-07-26T06:13:37.045692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:07.936415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241239+0000","last_change":"2024-07-26T06:13:34.066637+0000","last_active":"2024-07-26T06:13:36.241239+0000","last_peered":"2024-07-26T06:13:36.241239+0000","last_clean":"2024-07-26T06:13:36.241239+0000","last_became_active":"2024-07-26T06:13:34.066225+0000","last_became_peered":"2024-07-26T06:13:34.066225+0000","last_unstale":"2024-07-26T06:13:36.241239+0000","last_undegraded":"2024-07-26T06:13:36.241239+0000","last_fullsized":"2024-07-26T06:13:36.241239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165378+0000","last_change":"2024-07-26T06:13:35.080900+0000","last_active":"2024-07-26T06:13:36.165378+0000","last_peered":"2024-07-26T06:13:36.165378+0000","last_clean":"2024-07-26T06:13:36.165378+0000","last_became_active":"2024-07-26T06:13:35.080442+0000","last_became_peered":"2024-07-26T06:13:35.080442+0000","last_unstale":"2024-07-26T06:13:36.165378+0000","last_undegraded":"2024-07-26T06:13:36.165378+0000","last_fullsized":"2024-07-26T06:13:36.165378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165433+0000","last_change":"2024-07-26T06:13:34.060576+0000","last_active":"2024-07-26T06:13:36.165433+0000","last_peered":"2024-07-26T06:13:36.165433+0000","last_clean":"2024-07-26T06:13:36.165433+0000","last_became_active":"2024-07-26T06:13:34.060215+0000","last_became_peered":"2024-07-26T06:13:34.060215+0000","last_unstale":"2024-07-26T06:13:36.165433+0000","last_undegraded":"2024-07-26T06:13:36.165433+0000","last_fullsized":"2024-07-26T06:13:36.165433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:19.760539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057253+0000","last_change":"2024-07-26T06:13:35.072548+0000","last_active":"2024-07-26T06:13:36.057253+0000","last_peered":"2024-07-26T06:13:36.057253+0000","last_clean":"2024-07-26T06:13:36.057253+0000","last_became_active":"2024-07-26T06:13:35.072227+0000","last_became_peered":"2024-07-26T06:13:35.072227+0000","last_unstale":"2024-07-26T06:13:36.057253+0000","last_undegraded":"2024-07-26T06:13:36.057253+0000","last_fullsized":"2024-07-26T06:13:36.057253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046322+0000","last_change":"2024-07-26T06:13:34.064165+0000","last_active":"2024-07-26T06:13:37.046322+0000","last_peered":"2024-07-26T06:13:37.046322+0000","last_clean":"2024-07-26T06:13:37.046322+0000","last_became_active":"2024-07-26T06:13:34.063828+0000","last_became_peered":"2024-07-26T06:13:34.063828+0000","last_unstale":"2024-07-26T06:13:37.046322+0000","last_undegraded":"2024-07-26T06:13:37.046322+0000","last_fullsized":"2024-07-26T06:13:37.046322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:30.086773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063050+0000","last_change":"2024-07-26T06:13:35.064262+0000","last_active":"2024-07-26T06:13:36.063050+0000","last_peered":"2024-07-26T06:13:36.063050+0000","last_clean":"2024-07-26T06:13:36.063050+0000","last_became_active":"2024-07-26T06:13:35.061569+0000","last_became_peered":"2024-07-26T06:13:35.061569+0000","last_unstale":"2024-07-26T06:13:36.063050+0000","last_undegraded":"2024-07-26T06:13:36.063050+0000","last_fullsized":"2024-07-26T06:13:36.063050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:15:06.230764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241453+0000","last_change":"2024-07-26T06:13:34.053539+0000","last_active":"2024-07-26T06:13:36.241453+0000","last_peered":"2024-07-26T06:13:36.241453+0000","last_clean":"2024-07-26T06:13:36.241453+0000","last_became_active":"2024-07-26T06:13:34.053038+0000","last_became_peered":"2024-07-26T06:13:34.053038+0000","last_unstale":"2024-07-26T06:13:36.241453+0000","last_undegraded":"2024-07-26T06:13:36.241453+0000","last_fullsized":"2024-07-26T06:13:36.241453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267003+0000","last_change":"2024-07-26T06:13:35.075023+0000","last_active":"2024-07-26T06:13:37.267003+0000","last_peered":"2024-07-26T06:13:37.267003+0000","last_clean":"2024-07-26T06:13:37.267003+0000","last_became_active":"2024-07-26T06:13:35.073931+0000","last_became_peered":"2024-07-26T06:13:35.073931+0000","last_unstale":"2024-07-26T06:13:37.267003+0000","last_undegraded":"2024-07-26T06:13:37.267003+0000","last_fullsized":"2024-07-26T06:13:37.267003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059213+0000","last_change":"2024-07-26T06:13:34.048636+0000","last_active":"2024-07-26T06:13:36.059213+0000","last_peered":"2024-07-26T06:13:36.059213+0000","last_clean":"2024-07-26T06:13:36.059213+0000","last_became_active":"2024-07-26T06:13:34.048001+0000","last_became_peered":"2024-07-26T06:13:34.048001+0000","last_unstale":"2024-07-26T06:13:36.059213+0000","last_undegraded":"2024-07-26T06:13:36.059213+0000","last_fullsized":"2024-07-26T06:13:36.059213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:11:14.256545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.058850+0000","last_change":"2024-07-26T06:13:35.057953+0000","last_active":"2024-07-26T06:13:36.058850+0000","last_peered":"2024-07-26T06:13:36.058850+0000","last_clean":"2024-07-26T06:13:36.058850+0000","last_became_active":"2024-07-26T06:13:35.057762+0000","last_became_peered":"2024-07-26T06:13:35.057762+0000","last_unstale":"2024-07-26T06:13:36.058850+0000","last_undegraded":"2024-07-26T06:13:36.058850+0000","last_fullsized":"2024-07-26T06:13:36.058850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165279+0000","last_change":"2024-07-26T06:13:34.058750+0000","last_active":"2024-07-26T06:13:36.165279+0000","last_peered":"2024-07-26T06:13:36.165279+0000","last_clean":"2024-07-26T06:13:36.165279+0000","last_became_active":"2024-07-26T06:13:34.057317+0000","last_became_peered":"2024-07-26T06:13:34.057317+0000","last_unstale":"2024-07-26T06:13:36.165279+0000","last_undegraded":"2024-07-26T06:13:36.165279+0000","last_fullsized":"2024-07-26T06:13:36.165279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267198+0000","last_change":"2024-07-26T06:13:35.068556+0000","last_active":"2024-07-26T06:13:37.267198+0000","last_peered":"2024-07-26T06:13:37.267198+0000","last_clean":"2024-07-26T06:13:37.267198+0000","last_became_active":"2024-07-26T06:13:35.067965+0000","last_became_peered":"2024-07-26T06:13:35.067965+0000","last_unstale":"2024-07-26T06:13:37.267198+0000","last_undegraded":"2024-07-26T06:13:37.267198+0000","last_fullsized":"2024-07-26T06:13:37.267198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241420+0000","last_change":"2024-07-26T06:13:34.062439+0000","last_active":"2024-07-26T06:13:36.241420+0000","last_peered":"2024-07-26T06:13:36.241420+0000","last_clean":"2024-07-26T06:13:36.241420+0000","last_became_active":"2024-07-26T06:13:34.061547+0000","last_became_peered":"2024-07-26T06:13:34.061547+0000","last_unstale":"2024-07-26T06:13:36.241420+0000","last_undegraded":"2024-07-26T06:13:36.241420+0000","last_fullsized":"2024-07-26T06:13:36.241420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240885+0000","last_change":"2024-07-26T06:13:35.059120+0000","last_active":"2024-07-26T06:13:36.240885+0000","last_peered":"2024-07-26T06:13:36.240885+0000","last_clean":"2024-07-26T06:13:36.240885+0000","last_became_active":"2024-07-26T06:13:35.058851+0000","last_became_peered":"2024-07-26T06:13:35.058851+0000","last_unstale":"2024-07-26T06:13:36.240885+0000","last_undegraded":"2024-07-26T06:13:36.240885+0000","last_fullsized":"2024-07-26T06:13:36.240885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265520+0000","last_change":"2024-07-26T06:13:34.052072+0000","last_active":"2024-07-26T06:13:37.265520+0000","last_peered":"2024-07-26T06:13:37.265520+0000","last_clean":"2024-07-26T06:13:37.265520+0000","last_became_active":"2024-07-26T06:13:34.051775+0000","last_became_peered":"2024-07-26T06:13:34.051775+0000","last_unstale":"2024-07-26T06:13:37.265520+0000","last_undegraded":"2024-07-26T06:13:37.265520+0000","last_fullsized":"2024-07-26T06:13:37.265520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266571+0000","last_change":"2024-07-26T06:13:35.069871+0000","last_active":"2024-07-26T06:13:37.266571+0000","last_peered":"2024-07-26T06:13:37.266571+0000","last_clean":"2024-07-26T06:13:37.266571+0000","last_became_active":"2024-07-26T06:13:35.069272+0000","last_became_peered":"2024-07-26T06:13:35.069272+0000","last_unstale":"2024-07-26T06:13:37.266571+0000","last_undegraded":"2024-07-26T06:13:37.266571+0000","last_fullsized":"2024-07-26T06:13:37.266571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062736+0000","last_change":"2024-07-26T06:13:34.050786+0000","last_active":"2024-07-26T06:13:36.062736+0000","last_peered":"2024-07-26T06:13:36.062736+0000","last_clean":"2024-07-26T06:13:36.062736+0000","last_became_active":"2024-07-26T06:13:34.050291+0000","last_became_peered":"2024-07-26T06:13:34.050291+0000","last_unstale":"2024-07-26T06:13:36.062736+0000","last_undegraded":"2024-07-26T06:13:36.062736+0000","last_fullsized":"2024-07-26T06:13:36.062736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:28:47.174016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267602+0000","last_change":"2024-07-26T06:13:35.077176+0000","last_active":"2024-07-26T06:13:37.267602+0000","last_peered":"2024-07-26T06:13:37.267602+0000","last_clean":"2024-07-26T06:13:37.267602+0000","last_became_active":"2024-07-26T06:13:35.074703+0000","last_became_peered":"2024-07-26T06:13:35.074703+0000","last_unstale":"2024-07-26T06:13:37.267602+0000","last_undegraded":"2024-07-26T06:13:37.267602+0000","last_fullsized":"2024-07-26T06:13:37.267602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266222+0000","last_change":"2024-07-26T06:13:34.052622+0000","last_active":"2024-07-26T06:13:37.266222+0000","last_peered":"2024-07-26T06:13:37.266222+0000","last_clean":"2024-07-26T06:13:37.266222+0000","last_became_active":"2024-07-26T06:13:34.052311+0000","last_became_peered":"2024-07-26T06:13:34.052311+0000","last_unstale":"2024-07-26T06:13:37.266222+0000","last_undegraded":"2024-07-26T06:13:37.266222+0000","last_fullsized":"2024-07-26T06:13:37.266222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268227+0000","last_change":"2024-07-26T06:13:35.078765+0000","last_active":"2024-07-26T06:13:37.268227+0000","last_peered":"2024-07-26T06:13:37.268227+0000","last_clean":"2024-07-26T06:13:37.268227+0000","last_became_active":"2024-07-26T06:13:35.077968+0000","last_became_peered":"2024-07-26T06:13:35.077968+0000","last_unstale":"2024-07-26T06:13:37.268227+0000","last_undegraded":"2024-07-26T06:13:37.268227+0000","last_fullsized":"2024-07-26T06:13:37.268227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:22.947178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046400+0000","last_change":"2024-07-26T06:13:34.047184+0000","last_active":"2024-07-26T06:13:37.046400+0000","last_peered":"2024-07-26T06:13:37.046400+0000","last_clean":"2024-07-26T06:13:37.046400+0000","last_became_active":"2024-07-26T06:13:34.046868+0000","last_became_peered":"2024-07-26T06:13:34.046868+0000","last_unstale":"2024-07-26T06:13:37.046400+0000","last_undegraded":"2024-07-26T06:13:37.046400+0000","last_fullsized":"2024-07-26T06:13:37.046400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:17.313633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165826+0000","last_change":"2024-07-26T06:13:35.065849+0000","last_active":"2024-07-26T06:13:36.165826+0000","last_peered":"2024-07-26T06:13:36.165826+0000","last_clean":"2024-07-26T06:13:36.165826+0000","last_became_active":"2024-07-26T06:13:35.065135+0000","last_became_peered":"2024-07-26T06:13:35.065135+0000","last_unstale":"2024-07-26T06:13:36.165826+0000","last_undegraded":"2024-07-26T06:13:36.165826+0000","last_fullsized":"2024-07-26T06:13:36.165826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265761+0000","last_change":"2024-07-26T06:13:34.062785+0000","last_active":"2024-07-26T06:13:37.265761+0000","last_peered":"2024-07-26T06:13:37.265761+0000","last_clean":"2024-07-26T06:13:37.265761+0000","last_became_active":"2024-07-26T06:13:34.062466+0000","last_became_peered":"2024-07-26T06:13:34.062466+0000","last_unstale":"2024-07-26T06:13:37.265761+0000","last_undegraded":"2024-07-26T06:13:37.265761+0000","last_fullsized":"2024-07-26T06:13:37.265761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:13:51.856008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267453+0000","last_change":"2024-07-26T06:13:35.082619+0000","last_active":"2024-07-26T06:13:37.267453+0000","last_peered":"2024-07-26T06:13:37.267453+0000","last_clean":"2024-07-26T06:13:37.267453+0000","last_became_active":"2024-07-26T06:13:35.082044+0000","last_became_peered":"2024-07-26T06:13:35.082044+0000","last_unstale":"2024-07-26T06:13:37.267453+0000","last_undegraded":"2024-07-26T06:13:37.267453+0000","last_fullsized":"2024-07-26T06:13:37.267453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266471+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:13:37.266471+0000","last_peered":"2024-07-26T06:13:37.266471+0000","last_clean":"2024-07-26T06:13:37.266471+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:13:37.266471+0000","last_undegraded":"2024-07-26T06:13:37.266471+0000","last_fullsized":"2024-07-26T06:13:37.266471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164685+0000","last_change":"2024-07-26T06:13:34.061955+0000","last_active":"2024-07-26T06:13:36.164685+0000","last_peered":"2024-07-26T06:13:36.164685+0000","last_clean":"2024-07-26T06:13:36.164685+0000","last_became_active":"2024-07-26T06:13:34.060172+0000","last_became_peered":"2024-07-26T06:13:34.060172+0000","last_unstale":"2024-07-26T06:13:36.164685+0000","last_undegraded":"2024-07-26T06:13:36.164685+0000","last_fullsized":"2024-07-26T06:13:36.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:45.461779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059576+0000","last_change":"2024-07-26T06:13:35.077259+0000","last_active":"2024-07-26T06:13:36.059576+0000","last_peered":"2024-07-26T06:13:36.059576+0000","last_clean":"2024-07-26T06:13:36.059576+0000","last_became_active":"2024-07-26T06:13:35.076879+0000","last_became_peered":"2024-07-26T06:13:35.076879+0000","last_unstale":"2024-07-26T06:13:36.059576+0000","last_undegraded":"2024-07-26T06:13:36.059576+0000","last_fullsized":"2024-07-26T06:13:36.059576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:27.327738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266356+0000","last_change":"2024-07-26T06:13:34.064601+0000","last_active":"2024-07-26T06:13:37.266356+0000","last_peered":"2024-07-26T06:13:37.266356+0000","last_clean":"2024-07-26T06:13:37.266356+0000","last_became_active":"2024-07-26T06:13:34.064270+0000","last_became_peered":"2024-07-26T06:13:34.064270+0000","last_unstale":"2024-07-26T06:13:37.266356+0000","last_undegraded":"2024-07-26T06:13:37.266356+0000","last_fullsized":"2024-07-26T06:13:37.266356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056647+0000","last_change":"2024-07-26T06:13:35.062026+0000","last_active":"2024-07-26T06:13:36.056647+0000","last_peered":"2024-07-26T06:13:36.056647+0000","last_clean":"2024-07-26T06:13:36.056647+0000","last_became_active":"2024-07-26T06:13:35.061732+0000","last_became_peered":"2024-07-26T06:13:35.061732+0000","last_unstale":"2024-07-26T06:13:36.056647+0000","last_undegraded":"2024-07-26T06:13:36.056647+0000","last_fullsized":"2024-07-26T06:13:36.056647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:32.385097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265755+0000","last_change":"2024-07-26T06:13:34.057752+0000","last_active":"2024-07-26T06:13:37.265755+0000","last_peered":"2024-07-26T06:13:37.265755+0000","last_clean":"2024-07-26T06:13:37.265755+0000","last_became_active":"2024-07-26T06:13:34.057421+0000","last_became_peered":"2024-07-26T06:13:34.057421+0000","last_unstale":"2024-07-26T06:13:37.265755+0000","last_undegraded":"2024-07-26T06:13:37.265755+0000","last_fullsized":"2024-07-26T06:13:37.265755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045625+0000","last_change":"2024-07-26T06:13:35.051454+0000","last_active":"2024-07-26T06:13:37.045625+0000","last_peered":"2024-07-26T06:13:37.045625+0000","last_clean":"2024-07-26T06:13:37.045625+0000","last_became_active":"2024-07-26T06:13:35.051175+0000","last_became_peered":"2024-07-26T06:13:35.051175+0000","last_unstale":"2024-07-26T06:13:37.045625+0000","last_undegraded":"2024-07-26T06:13:37.045625+0000","last_fullsized":"2024-07-26T06:13:37.045625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:20.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165555+0000","last_change":"2024-07-26T06:13:34.055084+0000","last_active":"2024-07-26T06:13:36.165555+0000","last_peered":"2024-07-26T06:13:36.165555+0000","last_clean":"2024-07-26T06:13:36.165555+0000","last_became_active":"2024-07-26T06:13:34.054390+0000","last_became_peered":"2024-07-26T06:13:34.054390+0000","last_unstale":"2024-07-26T06:13:36.165555+0000","last_undegraded":"2024-07-26T06:13:36.165555+0000","last_fullsized":"2024-07-26T06:13:36.165555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062792+0000","last_change":"2024-07-26T06:13:35.052783+0000","last_active":"2024-07-26T06:13:36.062792+0000","last_peered":"2024-07-26T06:13:36.062792+0000","last_clean":"2024-07-26T06:13:36.062792+0000","last_became_active":"2024-07-26T06:13:35.048901+0000","last_became_peered":"2024-07-26T06:13:35.048901+0000","last_unstale":"2024-07-26T06:13:36.062792+0000","last_undegraded":"2024-07-26T06:13:36.062792+0000","last_fullsized":"2024-07-26T06:13:36.062792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:18.329878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045894+0000","last_change":"2024-07-26T06:13:34.049038+0000","last_active":"2024-07-26T06:13:37.045894+0000","last_peered":"2024-07-26T06:13:37.045894+0000","last_clean":"2024-07-26T06:13:37.045894+0000","last_became_active":"2024-07-26T06:13:34.048751+0000","last_became_peered":"2024-07-26T06:13:34.048751+0000","last_unstale":"2024-07-26T06:13:37.045894+0000","last_undegraded":"2024-07-26T06:13:37.045894+0000","last_fullsized":"2024-07-26T06:13:37.045894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165618+0000","last_change":"2024-07-26T06:13:35.080209+0000","last_active":"2024-07-26T06:13:36.165618+0000","last_peered":"2024-07-26T06:13:36.165618+0000","last_clean":"2024-07-26T06:13:36.165618+0000","last_became_active":"2024-07-26T06:13:35.079870+0000","last_became_peered":"2024-07-26T06:13:35.079870+0000","last_unstale":"2024-07-26T06:13:36.165618+0000","last_undegraded":"2024-07-26T06:13:36.165618+0000","last_fullsized":"2024-07-26T06:13:36.165618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268083+0000","last_change":"2024-07-26T06:13:34.066620+0000","last_active":"2024-07-26T06:13:37.268083+0000","last_peered":"2024-07-26T06:13:37.268083+0000","last_clean":"2024-07-26T06:13:37.268083+0000","last_became_active":"2024-07-26T06:13:34.066296+0000","last_became_peered":"2024-07-26T06:13:34.066296+0000","last_unstale":"2024-07-26T06:13:37.268083+0000","last_undegraded":"2024-07-26T06:13:37.268083+0000","last_fullsized":"2024-07-26T06:13:37.268083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:09:26.010192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045976+0000","last_change":"2024-07-26T06:13:35.052176+0000","last_active":"2024-07-26T06:13:37.045976+0000","last_peered":"2024-07-26T06:13:37.045976+0000","last_clean":"2024-07-26T06:13:37.045976+0000","last_became_active":"2024-07-26T06:13:35.051905+0000","last_became_peered":"2024-07-26T06:13:35.051905+0000","last_unstale":"2024-07-26T06:13:37.045976+0000","last_undegraded":"2024-07-26T06:13:37.045976+0000","last_fullsized":"2024-07-26T06:13:37.045976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:00.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046809+0000","last_change":"2024-07-26T06:13:34.059549+0000","last_active":"2024-07-26T06:13:37.046809+0000","last_peered":"2024-07-26T06:13:37.046809+0000","last_clean":"2024-07-26T06:13:37.046809+0000","last_became_active":"2024-07-26T06:13:34.058721+0000","last_became_peered":"2024-07-26T06:13:34.058721+0000","last_unstale":"2024-07-26T06:13:37.046809+0000","last_undegraded":"2024-07-26T06:13:37.046809+0000","last_fullsized":"2024-07-26T06:13:37.046809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060701+0000","last_change":"2024-07-26T06:13:35.066322+0000","last_active":"2024-07-26T06:13:36.060701+0000","last_peered":"2024-07-26T06:13:36.060701+0000","last_clean":"2024-07-26T06:13:36.060701+0000","last_became_active":"2024-07-26T06:13:35.066006+0000","last_became_peered":"2024-07-26T06:13:35.066006+0000","last_unstale":"2024-07-26T06:13:36.060701+0000","last_undegraded":"2024-07-26T06:13:36.060701+0000","last_fullsized":"2024-07-26T06:13:36.060701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:31.900907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241473+0000","last_change":"2024-07-26T06:13:34.069982+0000","last_active":"2024-07-26T06:13:36.241473+0000","last_peered":"2024-07-26T06:13:36.241473+0000","last_clean":"2024-07-26T06:13:36.241473+0000","last_became_active":"2024-07-26T06:13:34.069000+0000","last_became_peered":"2024-07-26T06:13:34.069000+0000","last_unstale":"2024-07-26T06:13:36.241473+0000","last_undegraded":"2024-07-26T06:13:36.241473+0000","last_fullsized":"2024-07-26T06:13:36.241473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059318+0000","last_change":"2024-07-26T06:13:35.059073+0000","last_active":"2024-07-26T06:13:36.059318+0000","last_peered":"2024-07-26T06:13:36.059318+0000","last_clean":"2024-07-26T06:13:36.059318+0000","last_became_active":"2024-07-26T06:13:35.058724+0000","last_became_peered":"2024-07-26T06:13:35.058724+0000","last_unstale":"2024-07-26T06:13:36.059318+0000","last_undegraded":"2024-07-26T06:13:36.059318+0000","last_fullsized":"2024-07-26T06:13:36.059318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266310+0000","last_change":"2024-07-26T06:13:34.065271+0000","last_active":"2024-07-26T06:13:37.266310+0000","last_peered":"2024-07-26T06:13:37.266310+0000","last_clean":"2024-07-26T06:13:37.266310+0000","last_became_active":"2024-07-26T06:13:34.064922+0000","last_became_peered":"2024-07-26T06:13:34.064922+0000","last_unstale":"2024-07-26T06:13:37.266310+0000","last_undegraded":"2024-07-26T06:13:37.266310+0000","last_fullsized":"2024-07-26T06:13:37.266310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062133+0000","last_change":"2024-07-26T06:13:35.056227+0000","last_active":"2024-07-26T06:13:36.062133+0000","last_peered":"2024-07-26T06:13:36.062133+0000","last_clean":"2024-07-26T06:13:36.062133+0000","last_became_active":"2024-07-26T06:13:35.055855+0000","last_became_peered":"2024-07-26T06:13:35.055855+0000","last_unstale":"2024-07-26T06:13:36.062133+0000","last_undegraded":"2024-07-26T06:13:36.062133+0000","last_fullsized":"2024-07-26T06:13:36.062133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:03.118475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045949+0000","last_change":"2024-07-26T06:13:34.066397+0000","last_active":"2024-07-26T06:13:37.045949+0000","last_peered":"2024-07-26T06:13:37.045949+0000","last_clean":"2024-07-26T06:13:37.045949+0000","last_became_active":"2024-07-26T06:13:34.066084+0000","last_became_peered":"2024-07-26T06:13:34.066084+0000","last_unstale":"2024-07-26T06:13:37.045949+0000","last_undegraded":"2024-07-26T06:13:37.045949+0000","last_fullsized":"2024-07-26T06:13:37.045949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:02.102230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059415+0000","last_change":"2024-07-26T06:13:35.053587+0000","last_active":"2024-07-26T06:13:36.059415+0000","last_peered":"2024-07-26T06:13:36.059415+0000","last_clean":"2024-07-26T06:13:36.059415+0000","last_became_active":"2024-07-26T06:13:35.053237+0000","last_became_peered":"2024-07-26T06:13:35.053237+0000","last_unstale":"2024-07-26T06:13:36.059415+0000","last_undegraded":"2024-07-26T06:13:36.059415+0000","last_fullsized":"2024-07-26T06:13:36.059415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:07.936415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046210+0000","last_change":"2024-07-26T06:13:34.044567+0000","last_active":"2024-07-26T06:13:37.046210+0000","last_peered":"2024-07-26T06:13:37.046210+0000","last_clean":"2024-07-26T06:13:37.046210+0000","last_became_active":"2024-07-26T06:13:34.043889+0000","last_became_peered":"2024-07-26T06:13:34.043889+0000","last_unstale":"2024-07-26T06:13:37.046210+0000","last_undegraded":"2024-07-26T06:13:37.046210+0000","last_fullsized":"2024-07-26T06:13:37.046210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:40.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060003+0000","last_change":"2024-07-26T06:13:35.043595+0000","last_active":"2024-07-26T06:13:36.060003+0000","last_peered":"2024-07-26T06:13:36.060003+0000","last_clean":"2024-07-26T06:13:36.060003+0000","last_became_active":"2024-07-26T06:13:35.043258+0000","last_became_peered":"2024-07-26T06:13:35.043258+0000","last_unstale":"2024-07-26T06:13:36.060003+0000","last_undegraded":"2024-07-26T06:13:36.060003+0000","last_fullsized":"2024-07-26T06:13:36.060003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:48.040021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267509+0000","last_change":"2024-07-26T06:13:34.064636+0000","last_active":"2024-07-26T06:13:37.267509+0000","last_peered":"2024-07-26T06:13:37.267509+0000","last_clean":"2024-07-26T06:13:37.267509+0000","last_became_active":"2024-07-26T06:13:34.064308+0000","last_became_peered":"2024-07-26T06:13:34.064308+0000","last_unstale":"2024-07-26T06:13:37.267509+0000","last_undegraded":"2024-07-26T06:13:37.267509+0000","last_fullsized":"2024-07-26T06:13:37.267509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240857+0000","last_change":"2024-07-26T06:13:35.066830+0000","last_active":"2024-07-26T06:13:36.240857+0000","last_peered":"2024-07-26T06:13:36.240857+0000","last_clean":"2024-07-26T06:13:36.240857+0000","last_became_active":"2024-07-26T06:13:35.066494+0000","last_became_peered":"2024-07-26T06:13:35.066494+0000","last_unstale":"2024-07-26T06:13:36.240857+0000","last_undegraded":"2024-07-26T06:13:36.240857+0000","last_fullsized":"2024-07-26T06:13:36.240857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165479+0000","last_change":"2024-07-26T06:13:35.079246+0000","last_active":"2024-07-26T06:13:36.165479+0000","last_peered":"2024-07-26T06:13:36.165479+0000","last_clean":"2024-07-26T06:13:36.165479+0000","last_became_active":"2024-07-26T06:13:35.075982+0000","last_became_peered":"2024-07-26T06:13:35.075982+0000","last_unstale":"2024-07-26T06:13:36.165479+0000","last_undegraded":"2024-07-26T06:13:36.165479+0000","last_fullsized":"2024-07-26T06:13:36.165479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267123+0000","last_change":"2024-07-26T06:13:34.066151+0000","last_active":"2024-07-26T06:13:37.267123+0000","last_peered":"2024-07-26T06:13:37.267123+0000","last_clean":"2024-07-26T06:13:37.267123+0000","last_became_active":"2024-07-26T06:13:34.065737+0000","last_became_peered":"2024-07-26T06:13:34.065737+0000","last_unstale":"2024-07-26T06:13:37.267123+0000","last_undegraded":"2024-07-26T06:13:37.267123+0000","last_fullsized":"2024-07-26T06:13:37.267123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165512+0000","last_change":"2024-07-26T06:13:35.075547+0000","last_active":"2024-07-26T06:13:36.165512+0000","last_peered":"2024-07-26T06:13:36.165512+0000","last_clean":"2024-07-26T06:13:36.165512+0000","last_became_active":"2024-07-26T06:13:35.075262+0000","last_became_peered":"2024-07-26T06:13:35.075262+0000","last_unstale":"2024-07-26T06:13:36.165512+0000","last_undegraded":"2024-07-26T06:13:36.165512+0000","last_fullsized":"2024-07-26T06:13:36.165512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267491+0000","last_change":"2024-07-26T06:13:34.066478+0000","last_active":"2024-07-26T06:13:37.267491+0000","last_peered":"2024-07-26T06:13:37.267491+0000","last_clean":"2024-07-26T06:13:37.267491+0000","last_became_active":"2024-07-26T06:13:34.066015+0000","last_became_peered":"2024-07-26T06:13:34.066015+0000","last_unstale":"2024-07-26T06:13:37.267491+0000","last_undegraded":"2024-07-26T06:13:37.267491+0000","last_fullsized":"2024-07-26T06:13:37.267491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:27.815076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059941+0000","last_change":"2024-07-26T06:13:35.077607+0000","last_active":"2024-07-26T06:13:36.059941+0000","last_peered":"2024-07-26T06:13:36.059941+0000","last_clean":"2024-07-26T06:13:36.059941+0000","last_became_active":"2024-07-26T06:13:35.077423+0000","last_became_peered":"2024-07-26T06:13:35.077423+0000","last_unstale":"2024-07-26T06:13:36.059941+0000","last_undegraded":"2024-07-26T06:13:36.059941+0000","last_fullsized":"2024-07-26T06:13:36.059941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060485+0000","last_change":"2024-07-26T06:13:34.052811+0000","last_active":"2024-07-26T06:13:36.060485+0000","last_peered":"2024-07-26T06:13:36.060485+0000","last_clean":"2024-07-26T06:13:36.060485+0000","last_became_active":"2024-07-26T06:13:34.051989+0000","last_became_peered":"2024-07-26T06:13:34.051989+0000","last_unstale":"2024-07-26T06:13:36.060485+0000","last_undegraded":"2024-07-26T06:13:36.060485+0000","last_fullsized":"2024-07-26T06:13:36.060485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060313+0000","last_change":"2024-07-26T06:13:35.061621+0000","last_active":"2024-07-26T06:13:36.060313+0000","last_peered":"2024-07-26T06:13:36.060313+0000","last_clean":"2024-07-26T06:13:36.060313+0000","last_became_active":"2024-07-26T06:13:35.061349+0000","last_became_peered":"2024-07-26T06:13:35.061349+0000","last_unstale":"2024-07-26T06:13:36.060313+0000","last_undegraded":"2024-07-26T06:13:36.060313+0000","last_fullsized":"2024-07-26T06:13:36.060313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:06:30.885224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059607+0000","last_change":"2024-07-26T06:13:34.069215+0000","last_active":"2024-07-26T06:13:36.059607+0000","last_peered":"2024-07-26T06:13:36.059607+0000","last_clean":"2024-07-26T06:13:36.059607+0000","last_became_active":"2024-07-26T06:13:34.068883+0000","last_became_peered":"2024-07-26T06:13:34.068883+0000","last_unstale":"2024-07-26T06:13:36.059607+0000","last_undegraded":"2024-07-26T06:13:36.059607+0000","last_fullsized":"2024-07-26T06:13:36.059607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:02:49.446347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":663491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509449,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":72541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":72667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:13:45.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db6600640 1 -- 172.21.15.163:0/1198651078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d98003a00 msgr2=0x7f2d98003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db6600640 1 --2- 172.21.15.163:0/1198651078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d98003a00 0x7f2d98003e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2da400baf0 tx=0x7f2d98005830 comp rx=0 tx=0).stop 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db5c00640 1 -- 172.21.15.163:0/1198651078 reap_dead start 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db6600640 1 -- 172.21.15.163:0/1198651078 shutdown_connections 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db6600640 1 -- 172.21.15.163:0/1198651078 >> 172.21.15.163:0/1198651078 conn(0x7f2db00cfda0 msgr2=0x7f2db00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db6600640 1 -- 172.21.15.163:0/1198651078 shutdown_connections 2024-07-26T06:13:45.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.573+0000 7f2db6600640 1 -- 172.21.15.163:0/1198651078 wait complete. 2024-07-26T06:13:45.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.575+0000 7f2db6600640 1 Processor -- start 2024-07-26T06:13:45.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.575+0000 7f2db6600640 1 -- start start 2024-07-26T06:13:45.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2db6600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d98003a00 0x7f2db00b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2db6600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 0x7f2db00cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2db6600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db0150260 0x7f2db01506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2db6600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2db00b6640 con 0x7f2db00cac70 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2db6600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2db00b64c0 con 0x7f2d98003a00 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2db6600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2db00b67c0 con 0x7f2db0150260 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2dafe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 0x7f2db00cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2dafe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 0x7f2db00cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36292/0 (socket says 172.21.15.163:36292) 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.576+0000 7f2dafe00640 1 -- 172.21.15.163:0/3130992447 learned_addr learned my addr 172.21.15.163:0/3130992447 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2db5200640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d98003a00 0x7f2db00b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2db5c00640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db0150260 0x7f2db01506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dafe00640 1 -- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db0150260 msgr2=0x7f2db01506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dafe00640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db0150260 0x7f2db01506e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dafe00640 1 -- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d98003a00 msgr2=0x7f2db00b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dafe00640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d98003a00 0x7f2db00b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dafe00640 1 -- 172.21.15.163:0/3130992447 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db01f4140 con 0x7f2db00cac70 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dafe00640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 0x7f2db00cb0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00d700 tx=0x7f2d9c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c017070 con 0x7f2db00cac70 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d9c009ab0 con 0x7f2db00cac70 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c010460 con 0x7f2db00cac70 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db00cb730 con 0x7f2db00cac70 2024-07-26T06:13:45.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.577+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db015f0b0 con 0x7f2db00cac70 2024-07-26T06:13:45.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.578+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db00ce2f0 con 0x7f2db00cac70 2024-07-26T06:13:45.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.580+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d9c01a070 con 0x7f2db00cac70 2024-07-26T06:13:45.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.580+0000 7f2dacc00640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2d90076300 0x7f2d900787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:45.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.581+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2d9c097770 con 0x7f2db00cac70 2024-07-26T06:13:45.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.581+0000 7f2db5200640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2d90076300 0x7f2d900787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:45.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.582+0000 7f2db5200640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2d90076300 0x7f2d900787c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2da4036000 tx=0x7f2da40046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:45.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.585+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d9c0610c0 con 0x7f2db00cac70 2024-07-26T06:13:45.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.769+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2db0159d20 con 0x7f2d90076300 2024-07-26T06:13:45.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.783+0000 7f2dacc00640 1 -- 172.21.15.163:0/3130992447 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251826 (secure 0 0 0) 0x7f2db0159d20 con 0x7f2d90076300 2024-07-26T06:13:45.788 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:45.790 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:13:45.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.789+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2d90076300 msgr2=0x7f2d900787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.789+0000 7f2db6600640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2d90076300 0x7f2d900787c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2da4036000 tx=0x7f2da40046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.789+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 msgr2=0x7f2db00cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:45.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.789+0000 7f2db6600640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 0x7f2db00cb0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00d700 tx=0x7f2d9c00dbc0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 shutdown_connections 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2d90076300 0x7f2d900787c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db0150260 0x7f2db01506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2db00cac70 0x7f2db00cb0f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 --2- 172.21.15.163:0/3130992447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d98003a00 0x7f2db00b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 >> 172.21.15.163:0/3130992447 conn(0x7f2db00cfda0 msgr2=0x7f2db015ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 shutdown_connections 2024-07-26T06:13:45.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:45.790+0000 7f2db6600640 1 -- 172.21.15.163:0/3130992447 wait complete. 2024-07-26T06:13:45.810 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-07-26T06:13:45.648010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219036,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755684,"statfs":{"total":773094113280,"available":772869820416,"internally_reserved":0,"allocated":2924544,"data_stored":1765884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004016"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267640+0000","last_change":"2024-07-26T06:13:35.072083+0000","last_active":"2024-07-26T06:13:37.267640+0000","last_peered":"2024-07-26T06:13:37.267640+0000","last_clean":"2024-07-26T06:13:37.267640+0000","last_became_active":"2024-07-26T06:13:35.071746+0000","last_became_peered":"2024-07-26T06:13:35.071746+0000","last_unstale":"2024-07-26T06:13:37.267640+0000","last_undegraded":"2024-07-26T06:13:37.267640+0000","last_fullsized":"2024-07-26T06:13:37.267640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046586+0000","last_change":"2024-07-26T06:13:35.058738+0000","last_active":"2024-07-26T06:13:37.046586+0000","last_peered":"2024-07-26T06:13:37.046586+0000","last_clean":"2024-07-26T06:13:37.046586+0000","last_became_active":"2024-07-26T06:13:35.058470+0000","last_became_peered":"2024-07-26T06:13:35.058470+0000","last_unstale":"2024-07-26T06:13:37.046586+0000","last_undegraded":"2024-07-26T06:13:37.046586+0000","last_fullsized":"2024-07-26T06:13:37.046586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:49.213504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056891+0000","last_change":"2024-07-26T06:13:35.042867+0000","last_active":"2024-07-26T06:13:36.056891+0000","last_peered":"2024-07-26T06:13:36.056891+0000","last_clean":"2024-07-26T06:13:36.056891+0000","last_became_active":"2024-07-26T06:13:35.042660+0000","last_became_peered":"2024-07-26T06:13:35.042660+0000","last_unstale":"2024-07-26T06:13:36.056891+0000","last_undegraded":"2024-07-26T06:13:36.056891+0000","last_fullsized":"2024-07-26T06:13:36.056891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:17:02.984714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165050+0000","last_change":"2024-07-26T06:13:35.076112+0000","last_active":"2024-07-26T06:13:36.165050+0000","last_peered":"2024-07-26T06:13:36.165050+0000","last_clean":"2024-07-26T06:13:36.165050+0000","last_became_active":"2024-07-26T06:13:35.075903+0000","last_became_peered":"2024-07-26T06:13:35.075903+0000","last_unstale":"2024-07-26T06:13:36.165050+0000","last_undegraded":"2024-07-26T06:13:36.165050+0000","last_fullsized":"2024-07-26T06:13:36.165050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267271+0000","last_change":"2024-07-26T06:13:35.068273+0000","last_active":"2024-07-26T06:13:37.267271+0000","last_peered":"2024-07-26T06:13:37.267271+0000","last_clean":"2024-07-26T06:13:37.267271+0000","last_became_active":"2024-07-26T06:13:35.067645+0000","last_became_peered":"2024-07-26T06:13:35.067645+0000","last_unstale":"2024-07-26T06:13:37.267271+0000","last_undegraded":"2024-07-26T06:13:37.267271+0000","last_fullsized":"2024-07-26T06:13:37.267271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164727+0000","last_change":"2024-07-26T06:13:35.080356+0000","last_active":"2024-07-26T06:13:36.164727+0000","last_peered":"2024-07-26T06:13:36.164727+0000","last_clean":"2024-07-26T06:13:36.164727+0000","last_became_active":"2024-07-26T06:13:35.079906+0000","last_became_peered":"2024-07-26T06:13:35.079906+0000","last_unstale":"2024-07-26T06:13:36.164727+0000","last_undegraded":"2024-07-26T06:13:36.164727+0000","last_fullsized":"2024-07-26T06:13:36.164727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266764+0000","last_change":"2024-07-26T06:13:35.070227+0000","last_active":"2024-07-26T06:13:37.266764+0000","last_peered":"2024-07-26T06:13:37.266764+0000","last_clean":"2024-07-26T06:13:37.266764+0000","last_became_active":"2024-07-26T06:13:35.069602+0000","last_became_peered":"2024-07-26T06:13:35.069602+0000","last_unstale":"2024-07-26T06:13:37.266764+0000","last_undegraded":"2024-07-26T06:13:37.266764+0000","last_fullsized":"2024-07-26T06:13:37.266764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059249+0000","last_change":"2024-07-26T06:13:35.077098+0000","last_active":"2024-07-26T06:13:36.059249+0000","last_peered":"2024-07-26T06:13:36.059249+0000","last_clean":"2024-07-26T06:13:36.059249+0000","last_became_active":"2024-07-26T06:13:35.076814+0000","last_became_peered":"2024-07-26T06:13:35.076814+0000","last_unstale":"2024-07-26T06:13:36.059249+0000","last_undegraded":"2024-07-26T06:13:36.059249+0000","last_fullsized":"2024-07-26T06:13:36.059249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:19.325249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267856+0000","last_change":"2024-07-26T06:13:35.081415+0000","last_active":"2024-07-26T06:13:37.267856+0000","last_peered":"2024-07-26T06:13:37.267856+0000","last_clean":"2024-07-26T06:13:37.267856+0000","last_became_active":"2024-07-26T06:13:35.080563+0000","last_became_peered":"2024-07-26T06:13:35.080563+0000","last_unstale":"2024-07-26T06:13:37.267856+0000","last_undegraded":"2024-07-26T06:13:37.267856+0000","last_fullsized":"2024-07-26T06:13:37.267856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241645+0000","last_change":"2024-07-26T06:13:35.067607+0000","last_active":"2024-07-26T06:13:36.241645+0000","last_peered":"2024-07-26T06:13:36.241645+0000","last_clean":"2024-07-26T06:13:36.241645+0000","last_became_active":"2024-07-26T06:13:35.064672+0000","last_became_peered":"2024-07-26T06:13:35.064672+0000","last_unstale":"2024-07-26T06:13:36.241645+0000","last_undegraded":"2024-07-26T06:13:36.241645+0000","last_fullsized":"2024-07-26T06:13:36.241645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241131+0000","last_change":"2024-07-26T06:13:35.070018+0000","last_active":"2024-07-26T06:13:36.241131+0000","last_peered":"2024-07-26T06:13:36.241131+0000","last_clean":"2024-07-26T06:13:36.241131+0000","last_became_active":"2024-07-26T06:13:35.069721+0000","last_became_peered":"2024-07-26T06:13:35.069721+0000","last_unstale":"2024-07-26T06:13:36.241131+0000","last_undegraded":"2024-07-26T06:13:36.241131+0000","last_fullsized":"2024-07-26T06:13:36.241131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063011+0000","last_change":"2024-07-26T06:13:35.062741+0000","last_active":"2024-07-26T06:13:36.063011+0000","last_peered":"2024-07-26T06:13:36.063011+0000","last_clean":"2024-07-26T06:13:36.063011+0000","last_became_active":"2024-07-26T06:13:35.061208+0000","last_became_peered":"2024-07-26T06:13:35.061208+0000","last_unstale":"2024-07-26T06:13:36.063011+0000","last_undegraded":"2024-07-26T06:13:36.063011+0000","last_fullsized":"2024-07-26T06:13:36.063011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:50:23.338780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059514+0000","last_change":"2024-07-26T06:13:35.061997+0000","last_active":"2024-07-26T06:13:36.059514+0000","last_peered":"2024-07-26T06:13:36.059514+0000","last_clean":"2024-07-26T06:13:36.059514+0000","last_became_active":"2024-07-26T06:13:35.061691+0000","last_became_peered":"2024-07-26T06:13:35.061691+0000","last_unstale":"2024-07-26T06:13:36.059514+0000","last_undegraded":"2024-07-26T06:13:36.059514+0000","last_fullsized":"2024-07-26T06:13:36.059514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:00.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056756+0000","last_change":"2024-07-26T06:13:35.060288+0000","last_active":"2024-07-26T06:13:36.056756+0000","last_peered":"2024-07-26T06:13:36.056756+0000","last_clean":"2024-07-26T06:13:36.056756+0000","last_became_active":"2024-07-26T06:13:35.059968+0000","last_became_peered":"2024-07-26T06:13:35.059968+0000","last_unstale":"2024-07-26T06:13:36.056756+0000","last_undegraded":"2024-07-26T06:13:36.056756+0000","last_fullsized":"2024-07-26T06:13:36.056756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:23:13.174113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045688+0000","last_change":"2024-07-26T06:13:35.063974+0000","last_active":"2024-07-26T06:13:37.045688+0000","last_peered":"2024-07-26T06:13:37.045688+0000","last_clean":"2024-07-26T06:13:37.045688+0000","last_became_active":"2024-07-26T06:13:35.062227+0000","last_became_peered":"2024-07-26T06:13:35.062227+0000","last_unstale":"2024-07-26T06:13:37.045688+0000","last_undegraded":"2024-07-26T06:13:37.045688+0000","last_fullsized":"2024-07-26T06:13:37.045688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:19.325249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165749+0000","last_change":"2024-07-26T06:13:35.079657+0000","last_active":"2024-07-26T06:13:36.165749+0000","last_peered":"2024-07-26T06:13:36.165749+0000","last_clean":"2024-07-26T06:13:36.165749+0000","last_became_active":"2024-07-26T06:13:35.078980+0000","last_became_peered":"2024-07-26T06:13:35.078980+0000","last_unstale":"2024-07-26T06:13:36.165749+0000","last_undegraded":"2024-07-26T06:13:36.165749+0000","last_fullsized":"2024-07-26T06:13:36.165749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:04.080870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165976+0000","last_change":"2024-07-26T06:13:35.081733+0000","last_active":"2024-07-26T06:13:36.165976+0000","last_peered":"2024-07-26T06:13:36.165976+0000","last_clean":"2024-07-26T06:13:36.165976+0000","last_became_active":"2024-07-26T06:13:35.081316+0000","last_became_peered":"2024-07-26T06:13:35.081316+0000","last_unstale":"2024-07-26T06:13:36.165976+0000","last_undegraded":"2024-07-26T06:13:36.165976+0000","last_fullsized":"2024-07-26T06:13:36.165976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266057+0000","last_change":"2024-07-26T06:13:35.062888+0000","last_active":"2024-07-26T06:13:37.266057+0000","last_peered":"2024-07-26T06:13:37.266057+0000","last_clean":"2024-07-26T06:13:37.266057+0000","last_became_active":"2024-07-26T06:13:35.062617+0000","last_became_peered":"2024-07-26T06:13:35.062617+0000","last_unstale":"2024-07-26T06:13:37.266057+0000","last_undegraded":"2024-07-26T06:13:37.266057+0000","last_fullsized":"2024-07-26T06:13:37.266057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:28.831320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268031+0000","last_change":"2024-07-26T06:13:35.073326+0000","last_active":"2024-07-26T06:13:37.268031+0000","last_peered":"2024-07-26T06:13:37.268031+0000","last_clean":"2024-07-26T06:13:37.268031+0000","last_became_active":"2024-07-26T06:13:35.073019+0000","last_became_peered":"2024-07-26T06:13:35.073019+0000","last_unstale":"2024-07-26T06:13:37.268031+0000","last_undegraded":"2024-07-26T06:13:37.268031+0000","last_fullsized":"2024-07-26T06:13:37.268031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:43:33.580626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241158+0000","last_change":"2024-07-26T06:13:35.067632+0000","last_active":"2024-07-26T06:13:36.241158+0000","last_peered":"2024-07-26T06:13:36.241158+0000","last_clean":"2024-07-26T06:13:36.241158+0000","last_became_active":"2024-07-26T06:13:35.066527+0000","last_became_peered":"2024-07-26T06:13:35.066527+0000","last_unstale":"2024-07-26T06:13:36.241158+0000","last_undegraded":"2024-07-26T06:13:36.241158+0000","last_fullsized":"2024-07-26T06:13:36.241158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057001+0000","last_change":"2024-07-26T06:13:35.056477+0000","last_active":"2024-07-26T06:13:36.057001+0000","last_peered":"2024-07-26T06:13:36.057001+0000","last_clean":"2024-07-26T06:13:36.057001+0000","last_became_active":"2024-07-26T06:13:35.056222+0000","last_became_peered":"2024-07-26T06:13:35.056222+0000","last_unstale":"2024-07-26T06:13:36.057001+0000","last_undegraded":"2024-07-26T06:13:36.057001+0000","last_fullsized":"2024-07-26T06:13:36.057001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:20.776784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164819+0000","last_change":"2024-07-26T06:13:35.081680+0000","last_active":"2024-07-26T06:13:36.164819+0000","last_peered":"2024-07-26T06:13:36.164819+0000","last_clean":"2024-07-26T06:13:36.164819+0000","last_became_active":"2024-07-26T06:13:35.081409+0000","last_became_peered":"2024-07-26T06:13:35.081409+0000","last_unstale":"2024-07-26T06:13:36.164819+0000","last_undegraded":"2024-07-26T06:13:36.164819+0000","last_fullsized":"2024-07-26T06:13:36.164819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:45:47.212513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059747+0000","last_change":"2024-07-26T06:13:35.063598+0000","last_active":"2024-07-26T06:13:36.059747+0000","last_peered":"2024-07-26T06:13:36.059747+0000","last_clean":"2024-07-26T06:13:36.059747+0000","last_became_active":"2024-07-26T06:13:35.063108+0000","last_became_peered":"2024-07-26T06:13:35.063108+0000","last_unstale":"2024-07-26T06:13:36.059747+0000","last_undegraded":"2024-07-26T06:13:36.059747+0000","last_fullsized":"2024-07-26T06:13:36.059747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:49.213504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164852+0000","last_change":"2024-07-26T06:13:35.078539+0000","last_active":"2024-07-26T06:13:36.164852+0000","last_peered":"2024-07-26T06:13:36.164852+0000","last_clean":"2024-07-26T06:13:36.164852+0000","last_became_active":"2024-07-26T06:13:35.074963+0000","last_became_peered":"2024-07-26T06:13:35.074963+0000","last_unstale":"2024-07-26T06:13:36.164852+0000","last_undegraded":"2024-07-26T06:13:36.164852+0000","last_fullsized":"2024-07-26T06:13:36.164852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:41.383992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060557+0000","last_change":"2024-07-26T06:13:35.053573+0000","last_active":"2024-07-26T06:13:36.060557+0000","last_peered":"2024-07-26T06:13:36.060557+0000","last_clean":"2024-07-26T06:13:36.060557+0000","last_became_active":"2024-07-26T06:13:35.053330+0000","last_became_peered":"2024-07-26T06:13:35.053330+0000","last_unstale":"2024-07-26T06:13:36.060557+0000","last_undegraded":"2024-07-26T06:13:36.060557+0000","last_fullsized":"2024-07-26T06:13:36.060557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:01.836541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241629+0000","last_change":"2024-07-26T06:13:35.074100+0000","last_active":"2024-07-26T06:13:36.241629+0000","last_peered":"2024-07-26T06:13:36.241629+0000","last_clean":"2024-07-26T06:13:36.241629+0000","last_became_active":"2024-07-26T06:13:35.073532+0000","last_became_peered":"2024-07-26T06:13:35.073532+0000","last_unstale":"2024-07-26T06:13:36.241629+0000","last_undegraded":"2024-07-26T06:13:36.241629+0000","last_fullsized":"2024-07-26T06:13:36.241629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046103+0000","last_change":"2024-07-26T06:13:35.053991+0000","last_active":"2024-07-26T06:13:37.046103+0000","last_peered":"2024-07-26T06:13:37.046103+0000","last_clean":"2024-07-26T06:13:37.046103+0000","last_became_active":"2024-07-26T06:13:35.053723+0000","last_became_peered":"2024-07-26T06:13:35.053723+0000","last_unstale":"2024-07-26T06:13:37.046103+0000","last_undegraded":"2024-07-26T06:13:37.046103+0000","last_fullsized":"2024-07-26T06:13:37.046103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:27.327738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060277+0000","last_change":"2024-07-26T06:13:35.055687+0000","last_active":"2024-07-26T06:13:36.060277+0000","last_peered":"2024-07-26T06:13:36.060277+0000","last_clean":"2024-07-26T06:13:36.060277+0000","last_became_active":"2024-07-26T06:13:35.054863+0000","last_became_peered":"2024-07-26T06:13:35.054863+0000","last_unstale":"2024-07-26T06:13:36.060277+0000","last_undegraded":"2024-07-26T06:13:36.060277+0000","last_fullsized":"2024-07-26T06:13:36.060277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:04.827844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059098+0000","last_change":"2024-07-26T06:13:35.047582+0000","last_active":"2024-07-26T06:13:36.059098+0000","last_peered":"2024-07-26T06:13:36.059098+0000","last_clean":"2024-07-26T06:13:36.059098+0000","last_became_active":"2024-07-26T06:13:35.047322+0000","last_became_peered":"2024-07-26T06:13:35.047322+0000","last_unstale":"2024-07-26T06:13:36.059098+0000","last_undegraded":"2024-07-26T06:13:36.059098+0000","last_fullsized":"2024-07-26T06:13:36.059098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:24.668717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165246+0000","last_change":"2024-07-26T06:13:35.079808+0000","last_active":"2024-07-26T06:13:36.165246+0000","last_peered":"2024-07-26T06:13:36.165246+0000","last_clean":"2024-07-26T06:13:36.165246+0000","last_became_active":"2024-07-26T06:13:35.076583+0000","last_became_peered":"2024-07-26T06:13:35.076583+0000","last_unstale":"2024-07-26T06:13:36.165246+0000","last_undegraded":"2024-07-26T06:13:36.165246+0000","last_fullsized":"2024-07-26T06:13:36.165246+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265681+0000","last_change":"2024-07-26T06:13:35.062887+0000","last_active":"2024-07-26T06:13:37.265681+0000","last_peered":"2024-07-26T06:13:37.265681+0000","last_clean":"2024-07-26T06:13:37.265681+0000","last_became_active":"2024-07-26T06:13:35.062553+0000","last_became_peered":"2024-07-26T06:13:35.062553+0000","last_unstale":"2024-07-26T06:13:37.265681+0000","last_undegraded":"2024-07-26T06:13:37.265681+0000","last_fullsized":"2024-07-26T06:13:37.265681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:38.675161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240837+0000","last_change":"2024-07-26T06:13:35.059916+0000","last_active":"2024-07-26T06:13:36.240837+0000","last_peered":"2024-07-26T06:13:36.240837+0000","last_clean":"2024-07-26T06:13:36.240837+0000","last_became_active":"2024-07-26T06:13:35.058276+0000","last_became_peered":"2024-07-26T06:13:35.058276+0000","last_unstale":"2024-07-26T06:13:36.240837+0000","last_undegraded":"2024-07-26T06:13:36.240837+0000","last_fullsized":"2024-07-26T06:13:36.240837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062155+0000","last_change":"2024-07-26T06:13:35.052170+0000","last_active":"2024-07-26T06:13:36.062155+0000","last_peered":"2024-07-26T06:13:36.062155+0000","last_clean":"2024-07-26T06:13:36.062155+0000","last_became_active":"2024-07-26T06:13:35.051630+0000","last_became_peered":"2024-07-26T06:13:35.051630+0000","last_unstale":"2024-07-26T06:13:36.062155+0000","last_undegraded":"2024-07-26T06:13:36.062155+0000","last_fullsized":"2024-07-26T06:13:36.062155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:31.103018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059899+0000","last_change":"2024-07-26T06:13:34.065590+0000","last_active":"2024-07-26T06:13:36.059899+0000","last_peered":"2024-07-26T06:13:36.059899+0000","last_clean":"2024-07-26T06:13:36.059899+0000","last_became_active":"2024-07-26T06:13:34.065275+0000","last_became_peered":"2024-07-26T06:13:34.065275+0000","last_unstale":"2024-07-26T06:13:36.059899+0000","last_undegraded":"2024-07-26T06:13:36.059899+0000","last_fullsized":"2024-07-26T06:13:36.059899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059022+0000","last_change":"2024-07-26T06:13:35.057161+0000","last_active":"2024-07-26T06:13:36.059022+0000","last_peered":"2024-07-26T06:13:36.059022+0000","last_clean":"2024-07-26T06:13:36.059022+0000","last_became_active":"2024-07-26T06:13:35.056843+0000","last_became_peered":"2024-07-26T06:13:35.056843+0000","last_unstale":"2024-07-26T06:13:36.059022+0000","last_undegraded":"2024-07-26T06:13:36.059022+0000","last_fullsized":"2024-07-26T06:13:36.059022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:59.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241143+0000","last_change":"2024-07-26T06:13:34.061026+0000","last_active":"2024-07-26T06:13:36.241143+0000","last_peered":"2024-07-26T06:13:36.241143+0000","last_clean":"2024-07-26T06:13:36.241143+0000","last_became_active":"2024-07-26T06:13:34.060729+0000","last_became_peered":"2024-07-26T06:13:34.060729+0000","last_unstale":"2024-07-26T06:13:36.241143+0000","last_undegraded":"2024-07-26T06:13:36.241143+0000","last_fullsized":"2024-07-26T06:13:36.241143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059737+0000","last_change":"2024-07-26T06:13:34.053732+0000","last_active":"2024-07-26T06:13:36.059737+0000","last_peered":"2024-07-26T06:13:36.059737+0000","last_clean":"2024-07-26T06:13:36.059737+0000","last_became_active":"2024-07-26T06:13:34.053385+0000","last_became_peered":"2024-07-26T06:13:34.053385+0000","last_unstale":"2024-07-26T06:13:36.059737+0000","last_undegraded":"2024-07-26T06:13:36.059737+0000","last_fullsized":"2024-07-26T06:13:36.059737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:24:03.458270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059243+0000","last_change":"2024-07-26T06:13:35.042064+0000","last_active":"2024-07-26T06:13:36.059243+0000","last_peered":"2024-07-26T06:13:36.059243+0000","last_clean":"2024-07-26T06:13:36.059243+0000","last_became_active":"2024-07-26T06:13:35.041741+0000","last_became_peered":"2024-07-26T06:13:35.041741+0000","last_unstale":"2024-07-26T06:13:36.059243+0000","last_undegraded":"2024-07-26T06:13:36.059243+0000","last_fullsized":"2024-07-26T06:13:36.059243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:20.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165112+0000","last_change":"2024-07-26T06:13:34.061502+0000","last_active":"2024-07-26T06:13:36.165112+0000","last_peered":"2024-07-26T06:13:36.165112+0000","last_clean":"2024-07-26T06:13:36.165112+0000","last_became_active":"2024-07-26T06:13:34.061155+0000","last_became_peered":"2024-07-26T06:13:34.061155+0000","last_unstale":"2024-07-26T06:13:36.165112+0000","last_undegraded":"2024-07-26T06:13:36.165112+0000","last_fullsized":"2024-07-26T06:13:36.165112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045692+0000","last_change":"2024-07-26T06:13:35.051729+0000","last_active":"2024-07-26T06:13:37.045692+0000","last_peered":"2024-07-26T06:13:37.045692+0000","last_clean":"2024-07-26T06:13:37.045692+0000","last_became_active":"2024-07-26T06:13:35.051429+0000","last_became_peered":"2024-07-26T06:13:35.051429+0000","last_unstale":"2024-07-26T06:13:37.045692+0000","last_undegraded":"2024-07-26T06:13:37.045692+0000","last_fullsized":"2024-07-26T06:13:37.045692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:07.936415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241239+0000","last_change":"2024-07-26T06:13:34.066637+0000","last_active":"2024-07-26T06:13:36.241239+0000","last_peered":"2024-07-26T06:13:36.241239+0000","last_clean":"2024-07-26T06:13:36.241239+0000","last_became_active":"2024-07-26T06:13:34.066225+0000","last_became_peered":"2024-07-26T06:13:34.066225+0000","last_unstale":"2024-07-26T06:13:36.241239+0000","last_undegraded":"2024-07-26T06:13:36.241239+0000","last_fullsized":"2024-07-26T06:13:36.241239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165378+0000","last_change":"2024-07-26T06:13:35.080900+0000","last_active":"2024-07-26T06:13:36.165378+0000","last_peered":"2024-07-26T06:13:36.165378+0000","last_clean":"2024-07-26T06:13:36.165378+0000","last_became_active":"2024-07-26T06:13:35.080442+0000","last_became_peered":"2024-07-26T06:13:35.080442+0000","last_unstale":"2024-07-26T06:13:36.165378+0000","last_undegraded":"2024-07-26T06:13:36.165378+0000","last_fullsized":"2024-07-26T06:13:36.165378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165433+0000","last_change":"2024-07-26T06:13:34.060576+0000","last_active":"2024-07-26T06:13:36.165433+0000","last_peered":"2024-07-26T06:13:36.165433+0000","last_clean":"2024-07-26T06:13:36.165433+0000","last_became_active":"2024-07-26T06:13:34.060215+0000","last_became_peered":"2024-07-26T06:13:34.060215+0000","last_unstale":"2024-07-26T06:13:36.165433+0000","last_undegraded":"2024-07-26T06:13:36.165433+0000","last_fullsized":"2024-07-26T06:13:36.165433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:19.760539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.057253+0000","last_change":"2024-07-26T06:13:35.072548+0000","last_active":"2024-07-26T06:13:36.057253+0000","last_peered":"2024-07-26T06:13:36.057253+0000","last_clean":"2024-07-26T06:13:36.057253+0000","last_became_active":"2024-07-26T06:13:35.072227+0000","last_became_peered":"2024-07-26T06:13:35.072227+0000","last_unstale":"2024-07-26T06:13:36.057253+0000","last_undegraded":"2024-07-26T06:13:36.057253+0000","last_fullsized":"2024-07-26T06:13:36.057253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046322+0000","last_change":"2024-07-26T06:13:34.064165+0000","last_active":"2024-07-26T06:13:37.046322+0000","last_peered":"2024-07-26T06:13:37.046322+0000","last_clean":"2024-07-26T06:13:37.046322+0000","last_became_active":"2024-07-26T06:13:34.063828+0000","last_became_peered":"2024-07-26T06:13:34.063828+0000","last_unstale":"2024-07-26T06:13:37.046322+0000","last_undegraded":"2024-07-26T06:13:37.046322+0000","last_fullsized":"2024-07-26T06:13:37.046322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:32:30.086773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.063050+0000","last_change":"2024-07-26T06:13:35.064262+0000","last_active":"2024-07-26T06:13:36.063050+0000","last_peered":"2024-07-26T06:13:36.063050+0000","last_clean":"2024-07-26T06:13:36.063050+0000","last_became_active":"2024-07-26T06:13:35.061569+0000","last_became_peered":"2024-07-26T06:13:35.061569+0000","last_unstale":"2024-07-26T06:13:36.063050+0000","last_undegraded":"2024-07-26T06:13:36.063050+0000","last_fullsized":"2024-07-26T06:13:36.063050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:15:06.230764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241453+0000","last_change":"2024-07-26T06:13:34.053539+0000","last_active":"2024-07-26T06:13:36.241453+0000","last_peered":"2024-07-26T06:13:36.241453+0000","last_clean":"2024-07-26T06:13:36.241453+0000","last_became_active":"2024-07-26T06:13:34.053038+0000","last_became_peered":"2024-07-26T06:13:34.053038+0000","last_unstale":"2024-07-26T06:13:36.241453+0000","last_undegraded":"2024-07-26T06:13:36.241453+0000","last_fullsized":"2024-07-26T06:13:36.241453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267003+0000","last_change":"2024-07-26T06:13:35.075023+0000","last_active":"2024-07-26T06:13:37.267003+0000","last_peered":"2024-07-26T06:13:37.267003+0000","last_clean":"2024-07-26T06:13:37.267003+0000","last_became_active":"2024-07-26T06:13:35.073931+0000","last_became_peered":"2024-07-26T06:13:35.073931+0000","last_unstale":"2024-07-26T06:13:37.267003+0000","last_undegraded":"2024-07-26T06:13:37.267003+0000","last_fullsized":"2024-07-26T06:13:37.267003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:22.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059213+0000","last_change":"2024-07-26T06:13:34.048636+0000","last_active":"2024-07-26T06:13:36.059213+0000","last_peered":"2024-07-26T06:13:36.059213+0000","last_clean":"2024-07-26T06:13:36.059213+0000","last_became_active":"2024-07-26T06:13:34.048001+0000","last_became_peered":"2024-07-26T06:13:34.048001+0000","last_unstale":"2024-07-26T06:13:36.059213+0000","last_undegraded":"2024-07-26T06:13:36.059213+0000","last_fullsized":"2024-07-26T06:13:36.059213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:11:14.256545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.058850+0000","last_change":"2024-07-26T06:13:35.057953+0000","last_active":"2024-07-26T06:13:36.058850+0000","last_peered":"2024-07-26T06:13:36.058850+0000","last_clean":"2024-07-26T06:13:36.058850+0000","last_became_active":"2024-07-26T06:13:35.057762+0000","last_became_peered":"2024-07-26T06:13:35.057762+0000","last_unstale":"2024-07-26T06:13:36.058850+0000","last_undegraded":"2024-07-26T06:13:36.058850+0000","last_fullsized":"2024-07-26T06:13:36.058850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:47.595416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165279+0000","last_change":"2024-07-26T06:13:34.058750+0000","last_active":"2024-07-26T06:13:36.165279+0000","last_peered":"2024-07-26T06:13:36.165279+0000","last_clean":"2024-07-26T06:13:36.165279+0000","last_became_active":"2024-07-26T06:13:34.057317+0000","last_became_peered":"2024-07-26T06:13:34.057317+0000","last_unstale":"2024-07-26T06:13:36.165279+0000","last_undegraded":"2024-07-26T06:13:36.165279+0000","last_fullsized":"2024-07-26T06:13:36.165279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267198+0000","last_change":"2024-07-26T06:13:35.068556+0000","last_active":"2024-07-26T06:13:37.267198+0000","last_peered":"2024-07-26T06:13:37.267198+0000","last_clean":"2024-07-26T06:13:37.267198+0000","last_became_active":"2024-07-26T06:13:35.067965+0000","last_became_peered":"2024-07-26T06:13:35.067965+0000","last_unstale":"2024-07-26T06:13:37.267198+0000","last_undegraded":"2024-07-26T06:13:37.267198+0000","last_fullsized":"2024-07-26T06:13:37.267198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:10.435518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241420+0000","last_change":"2024-07-26T06:13:34.062439+0000","last_active":"2024-07-26T06:13:36.241420+0000","last_peered":"2024-07-26T06:13:36.241420+0000","last_clean":"2024-07-26T06:13:36.241420+0000","last_became_active":"2024-07-26T06:13:34.061547+0000","last_became_peered":"2024-07-26T06:13:34.061547+0000","last_unstale":"2024-07-26T06:13:36.241420+0000","last_undegraded":"2024-07-26T06:13:36.241420+0000","last_fullsized":"2024-07-26T06:13:36.241420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240885+0000","last_change":"2024-07-26T06:13:35.059120+0000","last_active":"2024-07-26T06:13:36.240885+0000","last_peered":"2024-07-26T06:13:36.240885+0000","last_clean":"2024-07-26T06:13:36.240885+0000","last_became_active":"2024-07-26T06:13:35.058851+0000","last_became_peered":"2024-07-26T06:13:35.058851+0000","last_unstale":"2024-07-26T06:13:36.240885+0000","last_undegraded":"2024-07-26T06:13:36.240885+0000","last_fullsized":"2024-07-26T06:13:36.240885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265520+0000","last_change":"2024-07-26T06:13:34.052072+0000","last_active":"2024-07-26T06:13:37.265520+0000","last_peered":"2024-07-26T06:13:37.265520+0000","last_clean":"2024-07-26T06:13:37.265520+0000","last_became_active":"2024-07-26T06:13:34.051775+0000","last_became_peered":"2024-07-26T06:13:34.051775+0000","last_unstale":"2024-07-26T06:13:37.265520+0000","last_undegraded":"2024-07-26T06:13:37.265520+0000","last_fullsized":"2024-07-26T06:13:37.265520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266571+0000","last_change":"2024-07-26T06:13:35.069871+0000","last_active":"2024-07-26T06:13:37.266571+0000","last_peered":"2024-07-26T06:13:37.266571+0000","last_clean":"2024-07-26T06:13:37.266571+0000","last_became_active":"2024-07-26T06:13:35.069272+0000","last_became_peered":"2024-07-26T06:13:35.069272+0000","last_unstale":"2024-07-26T06:13:37.266571+0000","last_undegraded":"2024-07-26T06:13:37.266571+0000","last_fullsized":"2024-07-26T06:13:37.266571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:12.415581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062736+0000","last_change":"2024-07-26T06:13:34.050786+0000","last_active":"2024-07-26T06:13:36.062736+0000","last_peered":"2024-07-26T06:13:36.062736+0000","last_clean":"2024-07-26T06:13:36.062736+0000","last_became_active":"2024-07-26T06:13:34.050291+0000","last_became_peered":"2024-07-26T06:13:34.050291+0000","last_unstale":"2024-07-26T06:13:36.062736+0000","last_undegraded":"2024-07-26T06:13:36.062736+0000","last_fullsized":"2024-07-26T06:13:36.062736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:28:47.174016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267602+0000","last_change":"2024-07-26T06:13:35.077176+0000","last_active":"2024-07-26T06:13:37.267602+0000","last_peered":"2024-07-26T06:13:37.267602+0000","last_clean":"2024-07-26T06:13:37.267602+0000","last_became_active":"2024-07-26T06:13:35.074703+0000","last_became_peered":"2024-07-26T06:13:35.074703+0000","last_unstale":"2024-07-26T06:13:37.267602+0000","last_undegraded":"2024-07-26T06:13:37.267602+0000","last_fullsized":"2024-07-26T06:13:37.267602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266222+0000","last_change":"2024-07-26T06:13:34.052622+0000","last_active":"2024-07-26T06:13:37.266222+0000","last_peered":"2024-07-26T06:13:37.266222+0000","last_clean":"2024-07-26T06:13:37.266222+0000","last_became_active":"2024-07-26T06:13:34.052311+0000","last_became_peered":"2024-07-26T06:13:34.052311+0000","last_unstale":"2024-07-26T06:13:37.266222+0000","last_undegraded":"2024-07-26T06:13:37.266222+0000","last_fullsized":"2024-07-26T06:13:37.266222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:10.113455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268227+0000","last_change":"2024-07-26T06:13:35.078765+0000","last_active":"2024-07-26T06:13:37.268227+0000","last_peered":"2024-07-26T06:13:37.268227+0000","last_clean":"2024-07-26T06:13:37.268227+0000","last_became_active":"2024-07-26T06:13:35.077968+0000","last_became_peered":"2024-07-26T06:13:35.077968+0000","last_unstale":"2024-07-26T06:13:37.268227+0000","last_undegraded":"2024-07-26T06:13:37.268227+0000","last_fullsized":"2024-07-26T06:13:37.268227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:22.947178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046400+0000","last_change":"2024-07-26T06:13:34.047184+0000","last_active":"2024-07-26T06:13:37.046400+0000","last_peered":"2024-07-26T06:13:37.046400+0000","last_clean":"2024-07-26T06:13:37.046400+0000","last_became_active":"2024-07-26T06:13:34.046868+0000","last_became_peered":"2024-07-26T06:13:34.046868+0000","last_unstale":"2024-07-26T06:13:37.046400+0000","last_undegraded":"2024-07-26T06:13:37.046400+0000","last_fullsized":"2024-07-26T06:13:37.046400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:17.313633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165826+0000","last_change":"2024-07-26T06:13:35.065849+0000","last_active":"2024-07-26T06:13:36.165826+0000","last_peered":"2024-07-26T06:13:36.165826+0000","last_clean":"2024-07-26T06:13:36.165826+0000","last_became_active":"2024-07-26T06:13:35.065135+0000","last_became_peered":"2024-07-26T06:13:35.065135+0000","last_unstale":"2024-07-26T06:13:36.165826+0000","last_undegraded":"2024-07-26T06:13:36.165826+0000","last_fullsized":"2024-07-26T06:13:36.165826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:01.112465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265761+0000","last_change":"2024-07-26T06:13:34.062785+0000","last_active":"2024-07-26T06:13:37.265761+0000","last_peered":"2024-07-26T06:13:37.265761+0000","last_clean":"2024-07-26T06:13:37.265761+0000","last_became_active":"2024-07-26T06:13:34.062466+0000","last_became_peered":"2024-07-26T06:13:34.062466+0000","last_unstale":"2024-07-26T06:13:37.265761+0000","last_undegraded":"2024-07-26T06:13:37.265761+0000","last_fullsized":"2024-07-26T06:13:37.265761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:13:51.856008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267453+0000","last_change":"2024-07-26T06:13:35.082619+0000","last_active":"2024-07-26T06:13:37.267453+0000","last_peered":"2024-07-26T06:13:37.267453+0000","last_clean":"2024-07-26T06:13:37.267453+0000","last_became_active":"2024-07-26T06:13:35.082044+0000","last_became_peered":"2024-07-26T06:13:35.082044+0000","last_unstale":"2024-07-26T06:13:37.267453+0000","last_undegraded":"2024-07-26T06:13:37.267453+0000","last_fullsized":"2024-07-26T06:13:37.267453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266471+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:13:37.266471+0000","last_peered":"2024-07-26T06:13:37.266471+0000","last_clean":"2024-07-26T06:13:37.266471+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:13:37.266471+0000","last_undegraded":"2024-07-26T06:13:37.266471+0000","last_fullsized":"2024-07-26T06:13:37.266471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.164685+0000","last_change":"2024-07-26T06:13:34.061955+0000","last_active":"2024-07-26T06:13:36.164685+0000","last_peered":"2024-07-26T06:13:36.164685+0000","last_clean":"2024-07-26T06:13:36.164685+0000","last_became_active":"2024-07-26T06:13:34.060172+0000","last_became_peered":"2024-07-26T06:13:34.060172+0000","last_unstale":"2024-07-26T06:13:36.164685+0000","last_undegraded":"2024-07-26T06:13:36.164685+0000","last_fullsized":"2024-07-26T06:13:36.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:45.461779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059576+0000","last_change":"2024-07-26T06:13:35.077259+0000","last_active":"2024-07-26T06:13:36.059576+0000","last_peered":"2024-07-26T06:13:36.059576+0000","last_clean":"2024-07-26T06:13:36.059576+0000","last_became_active":"2024-07-26T06:13:35.076879+0000","last_became_peered":"2024-07-26T06:13:35.076879+0000","last_unstale":"2024-07-26T06:13:36.059576+0000","last_undegraded":"2024-07-26T06:13:36.059576+0000","last_fullsized":"2024-07-26T06:13:36.059576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:27.327738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266356+0000","last_change":"2024-07-26T06:13:34.064601+0000","last_active":"2024-07-26T06:13:37.266356+0000","last_peered":"2024-07-26T06:13:37.266356+0000","last_clean":"2024-07-26T06:13:37.266356+0000","last_became_active":"2024-07-26T06:13:34.064270+0000","last_became_peered":"2024-07-26T06:13:34.064270+0000","last_unstale":"2024-07-26T06:13:37.266356+0000","last_undegraded":"2024-07-26T06:13:37.266356+0000","last_fullsized":"2024-07-26T06:13:37.266356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.056647+0000","last_change":"2024-07-26T06:13:35.062026+0000","last_active":"2024-07-26T06:13:36.056647+0000","last_peered":"2024-07-26T06:13:36.056647+0000","last_clean":"2024-07-26T06:13:36.056647+0000","last_became_active":"2024-07-26T06:13:35.061732+0000","last_became_peered":"2024-07-26T06:13:35.061732+0000","last_unstale":"2024-07-26T06:13:36.056647+0000","last_undegraded":"2024-07-26T06:13:36.056647+0000","last_fullsized":"2024-07-26T06:13:36.056647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:32.385097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.265755+0000","last_change":"2024-07-26T06:13:34.057752+0000","last_active":"2024-07-26T06:13:37.265755+0000","last_peered":"2024-07-26T06:13:37.265755+0000","last_clean":"2024-07-26T06:13:37.265755+0000","last_became_active":"2024-07-26T06:13:34.057421+0000","last_became_peered":"2024-07-26T06:13:34.057421+0000","last_unstale":"2024-07-26T06:13:37.265755+0000","last_undegraded":"2024-07-26T06:13:37.265755+0000","last_fullsized":"2024-07-26T06:13:37.265755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:42:36.683066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045625+0000","last_change":"2024-07-26T06:13:35.051454+0000","last_active":"2024-07-26T06:13:37.045625+0000","last_peered":"2024-07-26T06:13:37.045625+0000","last_clean":"2024-07-26T06:13:37.045625+0000","last_became_active":"2024-07-26T06:13:35.051175+0000","last_became_peered":"2024-07-26T06:13:35.051175+0000","last_unstale":"2024-07-26T06:13:37.045625+0000","last_undegraded":"2024-07-26T06:13:37.045625+0000","last_fullsized":"2024-07-26T06:13:37.045625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:20.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165555+0000","last_change":"2024-07-26T06:13:34.055084+0000","last_active":"2024-07-26T06:13:36.165555+0000","last_peered":"2024-07-26T06:13:36.165555+0000","last_clean":"2024-07-26T06:13:36.165555+0000","last_became_active":"2024-07-26T06:13:34.054390+0000","last_became_peered":"2024-07-26T06:13:34.054390+0000","last_unstale":"2024-07-26T06:13:36.165555+0000","last_undegraded":"2024-07-26T06:13:36.165555+0000","last_fullsized":"2024-07-26T06:13:36.165555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062792+0000","last_change":"2024-07-26T06:13:35.052783+0000","last_active":"2024-07-26T06:13:36.062792+0000","last_peered":"2024-07-26T06:13:36.062792+0000","last_clean":"2024-07-26T06:13:36.062792+0000","last_became_active":"2024-07-26T06:13:35.048901+0000","last_became_peered":"2024-07-26T06:13:35.048901+0000","last_unstale":"2024-07-26T06:13:36.062792+0000","last_undegraded":"2024-07-26T06:13:36.062792+0000","last_fullsized":"2024-07-26T06:13:36.062792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:18.329878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045894+0000","last_change":"2024-07-26T06:13:34.049038+0000","last_active":"2024-07-26T06:13:37.045894+0000","last_peered":"2024-07-26T06:13:37.045894+0000","last_clean":"2024-07-26T06:13:37.045894+0000","last_became_active":"2024-07-26T06:13:34.048751+0000","last_became_peered":"2024-07-26T06:13:34.048751+0000","last_unstale":"2024-07-26T06:13:37.045894+0000","last_undegraded":"2024-07-26T06:13:37.045894+0000","last_fullsized":"2024-07-26T06:13:37.045894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165618+0000","last_change":"2024-07-26T06:13:35.080209+0000","last_active":"2024-07-26T06:13:36.165618+0000","last_peered":"2024-07-26T06:13:36.165618+0000","last_clean":"2024-07-26T06:13:36.165618+0000","last_became_active":"2024-07-26T06:13:35.079870+0000","last_became_peered":"2024-07-26T06:13:35.079870+0000","last_unstale":"2024-07-26T06:13:36.165618+0000","last_undegraded":"2024-07-26T06:13:36.165618+0000","last_fullsized":"2024-07-26T06:13:36.165618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:17.123979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.268083+0000","last_change":"2024-07-26T06:13:34.066620+0000","last_active":"2024-07-26T06:13:37.268083+0000","last_peered":"2024-07-26T06:13:37.268083+0000","last_clean":"2024-07-26T06:13:37.268083+0000","last_became_active":"2024-07-26T06:13:34.066296+0000","last_became_peered":"2024-07-26T06:13:34.066296+0000","last_unstale":"2024-07-26T06:13:37.268083+0000","last_undegraded":"2024-07-26T06:13:37.268083+0000","last_fullsized":"2024-07-26T06:13:37.268083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:09:26.010192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045976+0000","last_change":"2024-07-26T06:13:35.052176+0000","last_active":"2024-07-26T06:13:37.045976+0000","last_peered":"2024-07-26T06:13:37.045976+0000","last_clean":"2024-07-26T06:13:37.045976+0000","last_became_active":"2024-07-26T06:13:35.051905+0000","last_became_peered":"2024-07-26T06:13:35.051905+0000","last_unstale":"2024-07-26T06:13:37.045976+0000","last_undegraded":"2024-07-26T06:13:37.045976+0000","last_fullsized":"2024-07-26T06:13:37.045976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:00.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046809+0000","last_change":"2024-07-26T06:13:34.059549+0000","last_active":"2024-07-26T06:13:37.046809+0000","last_peered":"2024-07-26T06:13:37.046809+0000","last_clean":"2024-07-26T06:13:37.046809+0000","last_became_active":"2024-07-26T06:13:34.058721+0000","last_became_peered":"2024-07-26T06:13:34.058721+0000","last_unstale":"2024-07-26T06:13:37.046809+0000","last_undegraded":"2024-07-26T06:13:37.046809+0000","last_fullsized":"2024-07-26T06:13:37.046809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:20:28.324589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060701+0000","last_change":"2024-07-26T06:13:35.066322+0000","last_active":"2024-07-26T06:13:36.060701+0000","last_peered":"2024-07-26T06:13:36.060701+0000","last_clean":"2024-07-26T06:13:36.060701+0000","last_became_active":"2024-07-26T06:13:35.066006+0000","last_became_peered":"2024-07-26T06:13:35.066006+0000","last_unstale":"2024-07-26T06:13:36.060701+0000","last_undegraded":"2024-07-26T06:13:36.060701+0000","last_fullsized":"2024-07-26T06:13:36.060701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:31.900907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.241473+0000","last_change":"2024-07-26T06:13:34.069982+0000","last_active":"2024-07-26T06:13:36.241473+0000","last_peered":"2024-07-26T06:13:36.241473+0000","last_clean":"2024-07-26T06:13:36.241473+0000","last_became_active":"2024-07-26T06:13:34.069000+0000","last_became_peered":"2024-07-26T06:13:34.069000+0000","last_unstale":"2024-07-26T06:13:36.241473+0000","last_undegraded":"2024-07-26T06:13:36.241473+0000","last_fullsized":"2024-07-26T06:13:36.241473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:24:13.950920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059318+0000","last_change":"2024-07-26T06:13:35.059073+0000","last_active":"2024-07-26T06:13:36.059318+0000","last_peered":"2024-07-26T06:13:36.059318+0000","last_clean":"2024-07-26T06:13:36.059318+0000","last_became_active":"2024-07-26T06:13:35.058724+0000","last_became_peered":"2024-07-26T06:13:35.058724+0000","last_unstale":"2024-07-26T06:13:36.059318+0000","last_undegraded":"2024-07-26T06:13:36.059318+0000","last_fullsized":"2024-07-26T06:13:36.059318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:14:15.490285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.266310+0000","last_change":"2024-07-26T06:13:34.065271+0000","last_active":"2024-07-26T06:13:37.266310+0000","last_peered":"2024-07-26T06:13:37.266310+0000","last_clean":"2024-07-26T06:13:37.266310+0000","last_became_active":"2024-07-26T06:13:34.064922+0000","last_became_peered":"2024-07-26T06:13:34.064922+0000","last_unstale":"2024-07-26T06:13:37.266310+0000","last_undegraded":"2024-07-26T06:13:37.266310+0000","last_fullsized":"2024-07-26T06:13:37.266310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:49.142707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.062133+0000","last_change":"2024-07-26T06:13:35.056227+0000","last_active":"2024-07-26T06:13:36.062133+0000","last_peered":"2024-07-26T06:13:36.062133+0000","last_clean":"2024-07-26T06:13:36.062133+0000","last_became_active":"2024-07-26T06:13:35.055855+0000","last_became_peered":"2024-07-26T06:13:35.055855+0000","last_unstale":"2024-07-26T06:13:36.062133+0000","last_undegraded":"2024-07-26T06:13:36.062133+0000","last_fullsized":"2024-07-26T06:13:36.062133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:03.118475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.045949+0000","last_change":"2024-07-26T06:13:34.066397+0000","last_active":"2024-07-26T06:13:37.045949+0000","last_peered":"2024-07-26T06:13:37.045949+0000","last_clean":"2024-07-26T06:13:37.045949+0000","last_became_active":"2024-07-26T06:13:34.066084+0000","last_became_peered":"2024-07-26T06:13:34.066084+0000","last_unstale":"2024-07-26T06:13:37.045949+0000","last_undegraded":"2024-07-26T06:13:37.045949+0000","last_fullsized":"2024-07-26T06:13:37.045949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:33:02.102230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059415+0000","last_change":"2024-07-26T06:13:35.053587+0000","last_active":"2024-07-26T06:13:36.059415+0000","last_peered":"2024-07-26T06:13:36.059415+0000","last_clean":"2024-07-26T06:13:36.059415+0000","last_became_active":"2024-07-26T06:13:35.053237+0000","last_became_peered":"2024-07-26T06:13:35.053237+0000","last_unstale":"2024-07-26T06:13:36.059415+0000","last_undegraded":"2024-07-26T06:13:36.059415+0000","last_fullsized":"2024-07-26T06:13:36.059415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:07.936415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.046210+0000","last_change":"2024-07-26T06:13:34.044567+0000","last_active":"2024-07-26T06:13:37.046210+0000","last_peered":"2024-07-26T06:13:37.046210+0000","last_clean":"2024-07-26T06:13:37.046210+0000","last_became_active":"2024-07-26T06:13:34.043889+0000","last_became_peered":"2024-07-26T06:13:34.043889+0000","last_unstale":"2024-07-26T06:13:37.046210+0000","last_undegraded":"2024-07-26T06:13:37.046210+0000","last_fullsized":"2024-07-26T06:13:37.046210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:40.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060003+0000","last_change":"2024-07-26T06:13:35.043595+0000","last_active":"2024-07-26T06:13:36.060003+0000","last_peered":"2024-07-26T06:13:36.060003+0000","last_clean":"2024-07-26T06:13:36.060003+0000","last_became_active":"2024-07-26T06:13:35.043258+0000","last_became_peered":"2024-07-26T06:13:35.043258+0000","last_unstale":"2024-07-26T06:13:36.060003+0000","last_undegraded":"2024-07-26T06:13:36.060003+0000","last_fullsized":"2024-07-26T06:13:36.060003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:48.040021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267509+0000","last_change":"2024-07-26T06:13:34.064636+0000","last_active":"2024-07-26T06:13:37.267509+0000","last_peered":"2024-07-26T06:13:37.267509+0000","last_clean":"2024-07-26T06:13:37.267509+0000","last_became_active":"2024-07-26T06:13:34.064308+0000","last_became_peered":"2024-07-26T06:13:34.064308+0000","last_unstale":"2024-07-26T06:13:37.267509+0000","last_undegraded":"2024-07-26T06:13:37.267509+0000","last_fullsized":"2024-07-26T06:13:37.267509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:38:58.522497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.240857+0000","last_change":"2024-07-26T06:13:35.066830+0000","last_active":"2024-07-26T06:13:36.240857+0000","last_peered":"2024-07-26T06:13:36.240857+0000","last_clean":"2024-07-26T06:13:36.240857+0000","last_became_active":"2024-07-26T06:13:35.066494+0000","last_became_peered":"2024-07-26T06:13:35.066494+0000","last_unstale":"2024-07-26T06:13:36.240857+0000","last_undegraded":"2024-07-26T06:13:36.240857+0000","last_fullsized":"2024-07-26T06:13:36.240857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:59:52.909568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165479+0000","last_change":"2024-07-26T06:13:35.079246+0000","last_active":"2024-07-26T06:13:36.165479+0000","last_peered":"2024-07-26T06:13:36.165479+0000","last_clean":"2024-07-26T06:13:36.165479+0000","last_became_active":"2024-07-26T06:13:35.075982+0000","last_became_peered":"2024-07-26T06:13:35.075982+0000","last_unstale":"2024-07-26T06:13:36.165479+0000","last_undegraded":"2024-07-26T06:13:36.165479+0000","last_fullsized":"2024-07-26T06:13:36.165479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:02.576206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267123+0000","last_change":"2024-07-26T06:13:34.066151+0000","last_active":"2024-07-26T06:13:37.267123+0000","last_peered":"2024-07-26T06:13:37.267123+0000","last_clean":"2024-07-26T06:13:37.267123+0000","last_became_active":"2024-07-26T06:13:34.065737+0000","last_became_peered":"2024-07-26T06:13:34.065737+0000","last_unstale":"2024-07-26T06:13:37.267123+0000","last_undegraded":"2024-07-26T06:13:37.267123+0000","last_fullsized":"2024-07-26T06:13:37.267123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:10:37.658917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.165512+0000","last_change":"2024-07-26T06:13:35.075547+0000","last_active":"2024-07-26T06:13:36.165512+0000","last_peered":"2024-07-26T06:13:36.165512+0000","last_clean":"2024-07-26T06:13:36.165512+0000","last_became_active":"2024-07-26T06:13:35.075262+0000","last_became_peered":"2024-07-26T06:13:35.075262+0000","last_unstale":"2024-07-26T06:13:36.165512+0000","last_undegraded":"2024-07-26T06:13:36.165512+0000","last_fullsized":"2024-07-26T06:13:36.165512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:19.845599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:37.267491+0000","last_change":"2024-07-26T06:13:34.066478+0000","last_active":"2024-07-26T06:13:37.267491+0000","last_peered":"2024-07-26T06:13:37.267491+0000","last_clean":"2024-07-26T06:13:37.267491+0000","last_became_active":"2024-07-26T06:13:34.066015+0000","last_became_peered":"2024-07-26T06:13:34.066015+0000","last_unstale":"2024-07-26T06:13:37.267491+0000","last_undegraded":"2024-07-26T06:13:37.267491+0000","last_fullsized":"2024-07-26T06:13:37.267491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:27.815076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059941+0000","last_change":"2024-07-26T06:13:35.077607+0000","last_active":"2024-07-26T06:13:36.059941+0000","last_peered":"2024-07-26T06:13:36.059941+0000","last_clean":"2024-07-26T06:13:36.059941+0000","last_became_active":"2024-07-26T06:13:35.077423+0000","last_became_peered":"2024-07-26T06:13:35.077423+0000","last_unstale":"2024-07-26T06:13:36.059941+0000","last_undegraded":"2024-07-26T06:13:36.059941+0000","last_fullsized":"2024-07-26T06:13:36.059941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:46.478024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060485+0000","last_change":"2024-07-26T06:13:34.052811+0000","last_active":"2024-07-26T06:13:36.060485+0000","last_peered":"2024-07-26T06:13:36.060485+0000","last_clean":"2024-07-26T06:13:36.060485+0000","last_became_active":"2024-07-26T06:13:34.051989+0000","last_became_peered":"2024-07-26T06:13:34.051989+0000","last_unstale":"2024-07-26T06:13:36.060485+0000","last_undegraded":"2024-07-26T06:13:36.060485+0000","last_fullsized":"2024-07-26T06:13:36.060485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:46.722578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.060313+0000","last_change":"2024-07-26T06:13:35.061621+0000","last_active":"2024-07-26T06:13:36.060313+0000","last_peered":"2024-07-26T06:13:36.060313+0000","last_clean":"2024-07-26T06:13:36.060313+0000","last_became_active":"2024-07-26T06:13:35.061349+0000","last_became_peered":"2024-07-26T06:13:35.061349+0000","last_unstale":"2024-07-26T06:13:36.060313+0000","last_undegraded":"2024-07-26T06:13:36.060313+0000","last_fullsized":"2024-07-26T06:13:36.060313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:34.005944+0000","last_clean_scrub_stamp":"2024-07-26T06:13:34.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:06:30.885224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-07-26T06:13:36.059607+0000","last_change":"2024-07-26T06:13:34.069215+0000","last_active":"2024-07-26T06:13:36.059607+0000","last_peered":"2024-07-26T06:13:36.059607+0000","last_clean":"2024-07-26T06:13:36.059607+0000","last_became_active":"2024-07-26T06:13:34.068883+0000","last_became_peered":"2024-07-26T06:13:34.068883+0000","last_unstale":"2024-07-26T06:13:36.059607+0000","last_undegraded":"2024-07-26T06:13:36.059607+0000","last_fullsized":"2024-07-26T06:13:36.059607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:32.989700+0000","last_clean_scrub_stamp":"2024-07-26T06:13:32.989700+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:02:49.446347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":663491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":72631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509449,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":72541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":663547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":72667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":74153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:13:45.814 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:13:45.814 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T06:13:45.814 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-07-26T06:13:45.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.047+0000 7fb284a00640 1 -- 172.21.15.163:0/3779134316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb264004d40 msgr2=0x7fb2640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.047+0000 7fb284a00640 1 --2- 172.21.15.163:0/3779134316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb264004d40 0x7fb2640051c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c630 tx=0x7fb264005830 comp rx=0 tx=0).stop 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.048+0000 7fb27f200640 1 -- 172.21.15.163:0/3779134316 reap_dead start 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.048+0000 7fb284a00640 1 -- 172.21.15.163:0/3779134316 shutdown_connections 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.048+0000 7fb284a00640 1 -- 172.21.15.163:0/3779134316 >> 172.21.15.163:0/3779134316 conn(0x7fb2800b0e30 msgr2=0x7fb2800b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.048+0000 7fb284a00640 1 -- 172.21.15.163:0/3779134316 shutdown_connections 2024-07-26T06:13:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.048+0000 7fb284a00640 1 -- 172.21.15.163:0/3779134316 wait complete. 2024-07-26T06:13:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.049+0000 7fb284a00640 1 Processor -- start 2024-07-26T06:13:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.049+0000 7fb284a00640 1 -- start start 2024-07-26T06:13:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.050+0000 7fb284a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb264004d40 0x7fb2801574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.050+0000 7fb284a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 0x7fb2801e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.050+0000 7fb284a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2801e48f0 0x7fb28014f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.050+0000 7fb284a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2800718c0 con 0x7fb2801f66e0 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.050+0000 7fb284a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb280071740 con 0x7fb264004d40 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.050+0000 7fb284a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb2800b3e20 con 0x7fb2801e48f0 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 0x7fb2801e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 0x7fb2801e6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36314/0 (socket says 172.21.15.163:36314) 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 -- 172.21.15.163:0/2941626024 learned_addr learned my addr 172.21.15.163:0/2941626024 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 -- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2801e48f0 msgr2=0x7fb28014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2801e48f0 0x7fb28014f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb27e800640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb264004d40 0x7fb2801574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 -- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb264004d40 msgr2=0x7fb2801574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb264004d40 0x7fb2801574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 -- 172.21.15.163:0/2941626024 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb280157db0 con 0x7fb2801f66e0 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb275e00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 0x7fb2801e6fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb268010960 tx=0x7fb268010e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.051+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26800ece0 con 0x7fb2801f66e0 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.052+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb26800ee80 con 0x7fb2801f66e0 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.052+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268004270 con 0x7fb2801f66e0 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.052+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb280158090 con 0x7fb2801f66e0 2024-07-26T06:13:46.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.052+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb280159db0 con 0x7fb2801f66e0 2024-07-26T06:13:46.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.053+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb280157b10 con 0x7fb2801f66e0 2024-07-26T06:13:46.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.054+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb2680035f0 con 0x7fb2801f66e0 2024-07-26T06:13:46.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.055+0000 7fb277400640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb25c076300 0x7fb25c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.055+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb268098540 con 0x7fb2801f66e0 2024-07-26T06:13:46.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.055+0000 7fb27e800640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb25c076300 0x7fb25c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:46.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.056+0000 7fb27e800640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb25c076300 0x7fb25c0787c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb270000c00 tx=0x7fb2700069c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:46.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.059+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb268061e00 con 0x7fb2801f66e0 2024-07-26T06:13:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.290+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb2800b4380 con 0x7fb2801f66e0 2024-07-26T06:13:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.290+0000 7fb277400640 1 -- 172.21.15.163:0/2941626024 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb268061700 con 0x7fb2801f66e0 2024-07-26T06:13:46.291 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:46.291 INFO:teuthology.orchestra.run.smithi163.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T06:13:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.292+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb25c076300 msgr2=0x7fb25c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.292+0000 7fb284a00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb25c076300 0x7fb25c0787c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb270000c00 tx=0x7fb2700069c0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.292+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 msgr2=0x7fb2801e6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.292+0000 7fb284a00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 0x7fb2801e6fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb268010960 tx=0x7fb268010e20 comp rx=0 tx=0).stop 2024-07-26T06:13:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 shutdown_connections 2024-07-26T06:13:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb25c076300 0x7fb25c0787c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2801e48f0 0x7fb28014f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2801f66e0 0x7fb2801e6fc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 --2- 172.21.15.163:0/2941626024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb264004d40 0x7fb2801574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 >> 172.21.15.163:0/2941626024 conn(0x7fb2800b0e30 msgr2=0x7fb280152cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 shutdown_connections 2024-07-26T06:13:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.293+0000 7fb284a00640 1 -- 172.21.15.163:0/2941626024 wait complete. 2024-07-26T06:13:46.311 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-07-26T06:13:46.312 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.507+0000 7fe84ee00640 1 -- 172.21.15.163:0/140253360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 msgr2=0x7fe8480b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.507+0000 7fe84ee00640 1 --2- 172.21.15.163:0/140253360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe8480b2130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe8480cb230 tx=0x7fe84002aa30 comp rx=0 tx=0).stop 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.508+0000 7fe84ee00640 1 -- 172.21.15.163:0/140253360 shutdown_connections 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.508+0000 7fe84ee00640 1 --2- 172.21.15.163:0/140253360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe8480b2130 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.508+0000 7fe84ee00640 1 --2- 172.21.15.163:0/140253360 >> v2:172.21.15.164:3300/0 conn(0x7fe8480cbad0 0x7fe848159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.508+0000 7fe84ee00640 1 -- 172.21.15.163:0/140253360 >> 172.21.15.163:0/140253360 conn(0x7fe8480b0e30 msgr2=0x7fe8480b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.508+0000 7fe84ee00640 1 -- 172.21.15.163:0/140253360 shutdown_connections 2024-07-26T06:13:46.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.508+0000 7fe84ee00640 1 -- 172.21.15.163:0/140253360 wait complete. 2024-07-26T06:13:46.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.509+0000 7fe84ee00640 1 Processor -- start 2024-07-26T06:13:46.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.509+0000 7fe84ee00640 1 -- start start 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.510+0000 7fe84ee00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe848141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.510+0000 7fe84ee00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe848159370 0x7fe848158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.510+0000 7fe84ee00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe848142520 0x7fe848158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.510+0000 7fe84ee00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe84806f2f0 con 0x7fe8480b4420 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.510+0000 7fe84ee00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8480cd210 con 0x7fe848159370 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.510+0000 7fe84ee00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe84806f470 con 0x7fe848142520 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe848141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:46.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe848141fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36334/0 (socket says 172.21.15.163:36334) 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 -- 172.21.15.163:0/3255865812 learned_addr learned my addr 172.21.15.163:0/3255865812 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84d000640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe848159370 0x7fe848158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 -- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe848142520 msgr2=0x7fe848158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe848142520 0x7fe848158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 -- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe848159370 msgr2=0x7fe848158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe848159370 0x7fe848158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 -- 172.21.15.163:0/3255865812 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe848151090 con 0x7fe8480b4420 2024-07-26T06:13:46.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.511+0000 7fe84da00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe848141fe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b3f0 tx=0x7fe83c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:46.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.512+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c015020 con 0x7fe8480b4420 2024-07-26T06:13:46.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.512+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe83c011920 con 0x7fe8480b4420 2024-07-26T06:13:46.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.512+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c011d70 con 0x7fe8480b4420 2024-07-26T06:13:46.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.512+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe848151370 con 0x7fe8480b4420 2024-07-26T06:13:46.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.512+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84814da00 con 0x7fe8480b4420 2024-07-26T06:13:46.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.513+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8480b2130 con 0x7fe8480b4420 2024-07-26T06:13:46.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.514+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe83c018020 con 0x7fe8480b4420 2024-07-26T06:13:46.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.515+0000 7fe845600640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe828076300 0x7fe8280787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:46.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.515+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe83c0973c0 con 0x7fe8480b4420 2024-07-26T06:13:46.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.516+0000 7fe84d000640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe828076300 0x7fe8280787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:46.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.517+0000 7fe84d000640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe828076300 0x7fe8280787c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe8340058c0 tx=0x7fe834005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:46.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.519+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe83c01f030 con 0x7fe8480b4420 2024-07-26T06:13:46.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.770+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe848142f10 con 0x7fe8480b4420 2024-07-26T06:13:47.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.771+0000 7fe845600640 1 -- 172.21.15.163:0/3255865812 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7fe83c060d20 con 0x7fe8480b4420 2024-07-26T06:13:47.135 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:47.135 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:13:47.136 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:13:47.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe828076300 msgr2=0x7fe8280787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe828076300 0x7fe8280787c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe8340058c0 tx=0x7fe834005850 comp rx=0 tx=0).stop 2024-07-26T06:13:47.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 msgr2=0x7fe848141fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe848141fe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b3f0 tx=0x7fe83c00b8b0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 shutdown_connections 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe828076300 0x7fe8280787c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe848142520 0x7fe848158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe848159370 0x7fe848158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 --2- 172.21.15.163:0/3255865812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8480b4420 0x7fe848141fe0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 >> 172.21.15.163:0/3255865812 conn(0x7fe8480b0e30 msgr2=0x7fe8480b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 shutdown_connections 2024-07-26T06:13:47.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:46.774+0000 7fe84ee00640 1 -- 172.21.15.163:0/3255865812 wait complete. 2024-07-26T06:13:47.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.349+0000 7f0b90600640 1 -- 172.21.15.163:0/1392760932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b8815f4a0 msgr2=0x7f0b88070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.349+0000 7f0b90600640 1 --2- 172.21.15.163:0/1392760932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b8815f4a0 0x7f0b88070c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b80005fd0 tx=0x7f0b8002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.350+0000 7f0b90600640 1 -- 172.21.15.163:0/1392760932 shutdown_connections 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.350+0000 7f0b90600640 1 --2- 172.21.15.163:0/1392760932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b8815f4a0 0x7f0b88070c60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.350+0000 7f0b90600640 1 --2- 172.21.15.163:0/1392760932 >> v2:172.21.15.164:3300/0 conn(0x7f0b880b3770 0x7f0b8815ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.350+0000 7f0b90600640 1 -- 172.21.15.163:0/1392760932 >> 172.21.15.163:0/1392760932 conn(0x7f0b8806c600 msgr2=0x7f0b8806f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.350+0000 7f0b90600640 1 -- 172.21.15.163:0/1392760932 shutdown_connections 2024-07-26T06:13:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.350+0000 7f0b90600640 1 -- 172.21.15.163:0/1392760932 wait complete. 2024-07-26T06:13:47.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.351+0000 7f0b90600640 1 Processor -- start 2024-07-26T06:13:47.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 -- start start 2024-07-26T06:13:47.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b880b3770 0x7f0b88143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880b5fd0 0x7f0b8815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 0x7f0b8813bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0b88071530 con 0x7f0b8815f4a0 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880713b0 con 0x7f0b880b5fd0 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.352+0000 7f0b90600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0b880716b0 con 0x7f0b880b3770 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 0x7f0b8813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 0x7f0b8813bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36340/0 (socket says 172.21.15.163:36340) 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 -- 172.21.15.163:0/2643300755 learned_addr learned my addr 172.21.15.163:0/2643300755 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8dc00640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b880b3770 0x7f0b88143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8d200640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880b5fd0 0x7f0b8815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 -- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b880b3770 msgr2=0x7f0b88143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b880b3770 0x7f0b88143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 -- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880b5fd0 msgr2=0x7f0b8815cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880b5fd0 0x7f0b8815cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8e600640 1 -- 172.21.15.163:0/2643300755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8815bcc0 con 0x7f0b8815f4a0 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.353+0000 7f0b8d200640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880b5fd0 0x7f0b8815cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.354+0000 7f0b8e600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 0x7f0b8813bd10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0b80006010 tx=0x7f0b80031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.354+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80003e00 con 0x7f0b8815f4a0 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.354+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b80031a80 con 0x7f0b8815f4a0 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.354+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b800330d0 con 0x7f0b8815f4a0 2024-07-26T06:13:47.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.354+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8815bfa0 con 0x7f0b8815f4a0 2024-07-26T06:13:47.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.354+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b8815c560 con 0x7f0b8815f4a0 2024-07-26T06:13:47.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.355+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b88071ad0 con 0x7f0b8815f4a0 2024-07-26T06:13:47.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.356+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0b80033270 con 0x7f0b8815f4a0 2024-07-26T06:13:47.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.358+0000 7f0b85600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0b680762d0 0x7f0b68078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.358+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0b800b8420 con 0x7f0b8815f4a0 2024-07-26T06:13:47.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.359+0000 7f0b8dc00640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0b680762d0 0x7f0b68078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.360+0000 7f0b8dc00640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0b680762d0 0x7f0b68078790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0016b0 tx=0x7f0b7c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:47.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.362+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b8004a070 con 0x7f0b8815f4a0 2024-07-26T06:13:47.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.605+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b880b13c0 con 0x7f0b8815f4a0 2024-07-26T06:13:47.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.606+0000 7f0b85600640 1 -- 172.21.15.163:0/2643300755 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7f0b8003f030 con 0x7f0b8815f4a0 2024-07-26T06:13:47.607 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:47.607 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:13:47.607 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.608+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0b680762d0 msgr2=0x7f0b68078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.608+0000 7f0b90600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0b680762d0 0x7f0b68078790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0016b0 tx=0x7f0b7c009040 comp rx=0 tx=0).stop 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.608+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 msgr2=0x7f0b8813bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.608+0000 7f0b90600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 0x7f0b8813bd10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0b80006010 tx=0x7f0b80031650 comp rx=0 tx=0).stop 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 shutdown_connections 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0b680762d0 0x7f0b68078790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0b8815f4a0 0x7f0b8813bd10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880b5fd0 0x7f0b8815cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 --2- 172.21.15.163:0/2643300755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b880b3770 0x7f0b88143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 >> 172.21.15.163:0/2643300755 conn(0x7f0b8806c600 msgr2=0x7f0b880b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:47.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 shutdown_connections 2024-07-26T06:13:47.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.609+0000 7f0b90600640 1 -- 172.21.15.163:0/2643300755 wait complete. 2024-07-26T06:13:47.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.829+0000 7fe2db200640 1 -- 172.21.15.163:0/3169154436 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 msgr2=0x7fe2bc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.830+0000 7fe2db200640 1 --2- 172.21.15.163:0/3169154436 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2bc003e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009b50 tx=0x7fe2bc005830 comp rx=0 tx=0).stop 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.830+0000 7fe2da800640 1 -- 172.21.15.163:0/3169154436 reap_dead start 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.830+0000 7fe2db200640 1 -- 172.21.15.163:0/3169154436 shutdown_connections 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.830+0000 7fe2db200640 1 -- 172.21.15.163:0/3169154436 >> 172.21.15.163:0/3169154436 conn(0x7fe2dc0cfda0 msgr2=0x7fe2dc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.830+0000 7fe2db200640 1 -- 172.21.15.163:0/3169154436 shutdown_connections 2024-07-26T06:13:47.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.830+0000 7fe2db200640 1 -- 172.21.15.163:0/3169154436 wait complete. 2024-07-26T06:13:47.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.831+0000 7fe2db200640 1 Processor -- start 2024-07-26T06:13:47.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.832+0000 7fe2db200640 1 -- start start 2024-07-26T06:13:47.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.832+0000 7fe2db200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2dc14a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.832+0000 7fe2db200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2dc0c9060 0x7fe2dc0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.832+0000 7fe2db200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2dc0cbdb0 0x7fe2dc15d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2db200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc0ce680 con 0x7fe2bc003a30 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2db200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc0ce500 con 0x7fe2dc0cbdb0 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2db200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe2dc0b2740 con 0x7fe2dc0c9060 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2dc14a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2dc14a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36350/0 (socket says 172.21.15.163:36350) 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 -- 172.21.15.163:0/3028728380 learned_addr learned my addr 172.21.15.163:0/3028728380 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:47.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 -- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2dc0c9060 msgr2=0x7fe2dc0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2dc0c9060 0x7fe2dc0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2da800640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2dc0cbdb0 0x7fe2dc15d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 -- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2dc0cbdb0 msgr2=0x7fe2dc15d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2dc0cbdb0 0x7fe2dc15d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 -- 172.21.15.163:0/3028728380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2dc0ca7d0 con 0x7fe2bc003a30 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.833+0000 7fe2d9e00640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2dc14a830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc036040 tx=0x7fe2cc033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.834+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc043070 con 0x7fe2bc003a30 2024-07-26T06:13:47.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.834+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2cc004320 con 0x7fe2bc003a30 2024-07-26T06:13:47.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.834+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2dc0caab0 con 0x7fe2bc003a30 2024-07-26T06:13:47.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.834+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc038040 con 0x7fe2bc003a30 2024-07-26T06:13:47.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.834+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2dc0cb070 con 0x7fe2bc003a30 2024-07-26T06:13:47.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.835+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2dc0b64c0 con 0x7fe2bc003a30 2024-07-26T06:13:47.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.836+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe2cc046070 con 0x7fe2bc003a30 2024-07-26T06:13:47.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.838+0000 7fe2d2000640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe2b80762d0 0x7fe2b8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:47.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.838+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe2cc0b7b60 con 0x7fe2bc003a30 2024-07-26T06:13:47.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.838+0000 7fe2d9400640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe2b80762d0 0x7fe2b8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:47.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.840+0000 7fe2d9400640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe2b80762d0 0x7fe2b8078790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0cd770 tx=0x7fe2c4007cf0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:47.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:47.842+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe2cc080e60 con 0x7fe2bc003a30 2024-07-26T06:13:48.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.048+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2dc1583e0 con 0x7fe2bc003a30 2024-07-26T06:13:48.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.051+0000 7fe2d2000640 1 -- 172.21.15.163:0/3028728380 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7fe2cc085160 con 0x7fe2bc003a30 2024-07-26T06:13:48.052 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:48.052 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":21,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:36.040341+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:13:48.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe2b80762d0 msgr2=0x7fe2b8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe2b80762d0 0x7fe2b8078790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0cd770 tx=0x7fe2c4007cf0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 msgr2=0x7fe2dc14a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2dc14a830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc036040 tx=0x7fe2cc033670 comp rx=0 tx=0).stop 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 shutdown_connections 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe2b80762d0 0x7fe2b8078790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2dc0cbdb0 0x7fe2dc15d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2dc0c9060 0x7fe2dc0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 --2- 172.21.15.163:0/3028728380 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2bc003a30 0x7fe2dc14a830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.053+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 >> 172.21.15.163:0/3028728380 conn(0x7fe2dc0cfda0 msgr2=0x7fe2dc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.054+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 shutdown_connections 2024-07-26T06:13:48.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.054+0000 7fe2db200640 1 -- 172.21.15.163:0/3028728380 wait complete. 2024-07-26T06:13:48.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:13:48.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.271+0000 7faf97e00640 1 -- 172.21.15.163:0/3082809459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf98073820 msgr2=0x7faf98073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.271+0000 7faf97e00640 1 --2- 172.21.15.163:0/3082809459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf98073820 0x7faf98073cb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf980b4ac0 tx=0x7faf9002a980 comp rx=0 tx=0).stop 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.272+0000 7faf97e00640 1 -- 172.21.15.163:0/3082809459 shutdown_connections 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.272+0000 7faf97e00640 1 --2- 172.21.15.163:0/3082809459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf98073820 0x7faf98073cb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.272+0000 7faf97e00640 1 --2- 172.21.15.163:0/3082809459 >> v2:172.21.15.164:3300/0 conn(0x7faf980ae870 0x7faf980b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.272+0000 7faf97e00640 1 -- 172.21.15.163:0/3082809459 >> 172.21.15.163:0/3082809459 conn(0x7faf980b0e30 msgr2=0x7faf980b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.272+0000 7faf97e00640 1 -- 172.21.15.163:0/3082809459 shutdown_connections 2024-07-26T06:13:48.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.272+0000 7faf97e00640 1 -- 172.21.15.163:0/3082809459 wait complete. 2024-07-26T06:13:48.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.274+0000 7faf97e00640 1 Processor -- start 2024-07-26T06:13:48.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.274+0000 7faf97e00640 1 -- start start 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf98073820 0x7faf98158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf980ae870 0x7faf98145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 0x7faf9815cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faf9815a1d0 con 0x7faf980b3bd0 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faf9815a350 con 0x7faf980ae870 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf980741f0 con 0x7faf98073820 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.275+0000 7faf97400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 0x7faf9815cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:48.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 0x7faf9815cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36366/0 (socket says 172.21.15.163:36366) 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 -- 172.21.15.163:0/524745835 learned_addr learned my addr 172.21.15.163:0/524745835 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf96a00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf98073820 0x7faf98158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf8fe00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf980ae870 0x7faf98145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 -- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf98073820 msgr2=0x7faf98158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf98073820 0x7faf98158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 -- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf980ae870 msgr2=0x7faf98145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf980ae870 0x7faf98145a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 -- 172.21.15.163:0/524745835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf98157ca0 con 0x7faf980b3bd0 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97400640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 0x7faf9815cdb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf9002af70 tx=0x7faf90003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90034050 con 0x7faf980b3bd0 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf900045d0 con 0x7faf980b3bd0 2024-07-26T06:13:48.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90043440 con 0x7faf980b3bd0 2024-07-26T06:13:48.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.276+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf98157f80 con 0x7faf980b3bd0 2024-07-26T06:13:48.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.277+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf98158540 con 0x7faf980b3bd0 2024-07-26T06:13:48.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.278+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf98073cb0 con 0x7faf980b3bd0 2024-07-26T06:13:48.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.279+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faf90037020 con 0x7faf980b3bd0 2024-07-26T06:13:48.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.280+0000 7faf8ea00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faf740762d0 0x7faf74078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:48.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.280+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faf900b8c00 con 0x7faf980b3bd0 2024-07-26T06:13:48.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.280+0000 7faf96a00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faf740762d0 0x7faf74078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:48.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.281+0000 7faf96a00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faf740762d0 0x7faf74078790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf8800a730 tx=0x7faf88008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:48.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.284+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf900bf080 con 0x7faf980b3bd0 2024-07-26T06:13:48.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.519+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf980b26d0 con 0x7faf980b3bd0 2024-07-26T06:13:48.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.520+0000 7faf8ea00640 1 -- 172.21.15.163:0/524745835 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7faf90082560 con 0x7faf980b3bd0 2024-07-26T06:13:48.523 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:13:48.523 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:13:48.524 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:13:48.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faf740762d0 msgr2=0x7faf74078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faf740762d0 0x7faf74078790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf8800a730 tx=0x7faf88008040 comp rx=0 tx=0).stop 2024-07-26T06:13:48.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 msgr2=0x7faf9815cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:48.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 0x7faf9815cdb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf9002af70 tx=0x7faf90003f70 comp rx=0 tx=0).stop 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 shutdown_connections 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faf740762d0 0x7faf74078790 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf980b3bd0 0x7faf9815cdb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf980ae870 0x7faf98145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 --2- 172.21.15.163:0/524745835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf98073820 0x7faf98158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 >> 172.21.15.163:0/524745835 conn(0x7faf980b0e30 msgr2=0x7faf980ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 shutdown_connections 2024-07-26T06:13:48.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:48.523+0000 7faf97e00640 1 -- 172.21.15.163:0/524745835 wait complete. 2024-07-26T06:13:48.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:13:34.510631+0000', 'modified': '2024-07-26T06:13:37.550193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4372}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4372': {'gid': 4372, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.163:6835/2923632545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6834', 'nonce': 2923632545}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 2923632545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4372, 'qdb_cluster': [4372]} 2024-07-26T06:13:48.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:13:48.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-07-26T06:13:49.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e68c00640 1 -- 172.21.15.163:0/4190738169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e48004800 msgr2=0x7f5e48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e68c00640 1 --2- 172.21.15.163:0/4190738169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e48004800 0x7f5e48004c80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5e50006fb0 tx=0x7f5e48005830 comp rx=0 tx=0).stop 2024-07-26T06:13:49.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e63e00640 1 -- 172.21.15.163:0/4190738169 reap_dead start 2024-07-26T06:13:49.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e68c00640 1 -- 172.21.15.163:0/4190738169 shutdown_connections 2024-07-26T06:13:49.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e68c00640 1 -- 172.21.15.163:0/4190738169 >> 172.21.15.163:0/4190738169 conn(0x7f5e640b0e30 msgr2=0x7f5e640b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:49.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e68c00640 1 -- 172.21.15.163:0/4190738169 shutdown_connections 2024-07-26T06:13:49.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.156+0000 7f5e68c00640 1 -- 172.21.15.163:0/4190738169 wait complete. 2024-07-26T06:13:49.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.158+0000 7f5e68c00640 1 Processor -- start 2024-07-26T06:13:49.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.158+0000 7f5e68c00640 1 -- start start 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e68c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e48004800 0x7f5e64145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e68c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e64156690 0x7f5e6414f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e68c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 0x7f5e641f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e68c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5e640cb9f0 con 0x7f5e641f3bf0 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e68c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5e640cb870 con 0x7f5e64156690 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e68c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6415a1d0 con 0x7f5e48004800 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 0x7f5e641f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 0x7f5e641f4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36392/0 (socket says 172.21.15.163:36392) 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 -- 172.21.15.163:0/237533312 learned_addr learned my addr 172.21.15.163:0/237533312 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e62a00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e64156690 0x7f5e6414f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63400640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e48004800 0x7f5e64145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 -- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e48004800 msgr2=0x7f5e64145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e48004800 0x7f5e64145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 -- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e64156690 msgr2=0x7f5e6414f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e64156690 0x7f5e6414f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.159+0000 7f5e63e00640 1 -- 172.21.15.163:0/237533312 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e64156b90 con 0x7f5e641f3bf0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.160+0000 7f5e63e00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 0x7f5e641f4080 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e48004780 tx=0x7f5e5c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.160+0000 7f5e5b400640 1 -- 172.21.15.163:0/237533312 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c00cc30 con 0x7f5e641f3bf0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.160+0000 7f5e5b400640 1 -- 172.21.15.163:0/237533312 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e5c019070 con 0x7f5e641f3bf0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.160+0000 7f5e5b400640 1 -- 172.21.15.163:0/237533312 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c0116d0 con 0x7f5e641f3bf0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.160+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e64156e70 con 0x7f5e641f3bf0 2024-07-26T06:13:49.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.160+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e64157430 con 0x7f5e641f3bf0 2024-07-26T06:13:49.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.162+0000 7f5e5b400640 1 -- 172.21.15.163:0/237533312 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5e5c011870 con 0x7f5e641f3bf0 2024-07-26T06:13:49.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.163+0000 7f5e5b400640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5e44076300 0x7f5e440787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.164+0000 7f5e5b400640 1 -- 172.21.15.163:0/237533312 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5e5c097b90 con 0x7f5e641f3bf0 2024-07-26T06:13:49.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.164+0000 7f5e6cf88740 1 Processor -- start 2024-07-26T06:13:49.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.164+0000 7f5e6cf88740 1 -- start start 2024-07-26T06:13:49.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.164+0000 7f5e63400640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5e44076300 0x7f5e440787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db283bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283be8d0 0x55db283bcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 0x55db28372fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db2828f3f0 con 0x55db28379e50 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55db282ea5c0 con 0x55db283be8d0 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55db282d9b60 con 0x55db283bf560 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e63e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 0x55db28372fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e63400640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5e44076300 0x7f5e440787c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009b20 tx=0x7f5e54005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e63e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 0x55db28372fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36394/0 (socket says 172.21.15.163:36394) 2024-07-26T06:13:49.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.165+0000 7f5e63e00640 1 -- 172.21.15.163:0/3480526621 learned_addr learned my addr 172.21.15.163:0/3480526621 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63400640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db283bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e62a00640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283be8d0 0x55db283bcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63e00640 1 -- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 msgr2=0x55db283bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63e00640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db283bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63e00640 1 -- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283be8d0 msgr2=0x55db283bcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63e00640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283be8d0 0x55db283bcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63e00640 1 -- 172.21.15.163:0/3480526621 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2803e850 con 0x55db28379e50 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63400640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db283bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e62a00640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283be8d0 0x55db283bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e63e00640 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 0x55db28372fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c060ce0 tx=0x7f5e5c060be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a411b5027e094d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e4f400640 1 -- 172.21.15.163:0/3480526621 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c061450 con 0x55db28379e50 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e4f400640 1 -- 172.21.15.163:0/3480526621 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e5c060d50 con 0x55db28379e50 2024-07-26T06:13:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.166+0000 7f5e4f400640 1 -- 172.21.15.163:0/3480526621 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c011d80 con 0x55db28379e50 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 -- 172.21.15.163:0/3480526621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 msgr2=0x55db28372fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 0x55db28372fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c060ce0 tx=0x7f5e5c060be0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 -- 172.21.15.163:0/3480526621 shutdown_connections 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db28379e50 0x55db28372fd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283be8d0 0x55db283bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/3480526621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db283bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 -- 172.21.15.163:0/3480526621 >> 172.21.15.163:0/3480526621 conn(0x55db283751a0 msgr2=0x55db283bd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:49.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 -- 172.21.15.163:0/3480526621 shutdown_connections 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.167+0000 7f5e6cf88740 1 -- 172.21.15.163:0/3480526621 wait complete. 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.168+0000 7f5e6cf88740 1 -- start start 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.168+0000 7f5e6cf88740 1 Processor -- start 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28394130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db283b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283b7bc0 0x55db283b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db28282210 con 0x55db2839f270 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55db282ddba0 con 0x55db283b7bc0 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55db283bdce0 con 0x55db283bf560 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e6cf88740 10 client.0 _ll_register_callbacks cb 0x55db282828a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db283b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db283b7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36400/0 (socket says 172.21.15.163:36400) 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 -- 172.21.15.163:0/1770069252 learned_addr learned my addr 172.21.15.163:0/1770069252 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e63400640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28394130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e63e00640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283b7bc0 0x55db283b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 -- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 msgr2=0x55db28394130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28394130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 -- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283b7bc0 msgr2=0x55db283b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283b7bc0 0x55db283b5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.169+0000 7f5e62a00640 1 -- 172.21.15.163:0/1770069252 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db280b56b0 con 0x55db2839f270 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e62a00640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db283b7680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5e5002b0b0 tx=0x7f5e50001d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5de4017cd0e9d46 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e43e00640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e43e00640 1 -- 172.21.15.163:0/1770069252 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50031040 con 0x55db2839f270 2024-07-26T06:13:49.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e43e00640 1 -- 172.21.15.163:0/1770069252 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5e50043b00 con 0x55db2839f270 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55db282ec510 con 0x55db2839f270 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e43e00640 1 -- 172.21.15.163:0/1770069252 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f5e50043d00 con 0x55db2839f270 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e6cf88740 10 client.4638 fetch_fsmap learned FSMap version 8 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.170+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55db283a2cc0 con 0x55db2839f270 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e43e00640 1 -- 172.21.15.163:0/1770069252 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55db282ec510 con 0x55db2839f270 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e43e00640 1 -- 172.21.15.163:0/1770069252 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f5e50006050 con 0x55db2839f270 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e6cf88740 10 client.4638 fetch_fsmap finished waiting for FSMap version 8 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e6cf88740 10 client.4638 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e6cf88740 20 client.4638 populate_metadata read hostname 'smithi163' 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db283ba620 0x55db283dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e6cf88740 4 client.4638 mds_command: new command op to 4372 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 --> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db282e7fb0 con 0x55db283ba620 2024-07-26T06:13:49.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.171+0000 7f5e63400640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db283ba620 0x55db283dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.172+0000 7f5e63400640 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db283ba620 0x55db283dba70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c8c960a316c2411 server_cookie=33fd08d982ac93ff in_seq=0 out_seq=0 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.172+0000 7f5e43e00640 10 client.4638 ms_handle_connect on v2:172.21.15.163:6834/2923632545 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.172+0000 7f5e43e00640 1 -- 172.21.15.163:0/1770069252 <== mds.0 v2:172.21.15.163:6834/2923632545 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55db282e7fb0 con 0x55db283ba620 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.172+0000 7f5e43e00640 10 client.4638 handle_command_reply: tid=0 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.172+0000 7f5e6cf88740 1 client.4638 shutdown 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 msgr2=0x55db283b7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db283b7680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5e5002b0b0 tx=0x7f5e50001d30 comp rx=0 tx=0).stop 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 shutdown_connections 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db283ba620 0x55db283dba70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283b7bc0 0x55db283b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db283b7680 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1770069252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28394130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e43e00640 0 client.4638 ms_handle_reset on v2:172.21.15.163:6834/2923632545 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 >> 172.21.15.163:0/1770069252 conn(0x55db28379e50 msgr2=0x55db28378c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 shutdown_connections 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1770069252 wait complete. 2024-07-26T06:13:49.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.173+0000 7f5e6cf88740 20 client.4638 trim_cache size 0 max 16384 2024-07-26T06:13:49.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.193+0000 7f5e6cf88740 1 Processor -- start 2024-07-26T06:13:49.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.193+0000 7f5e6cf88740 1 -- start start 2024-07-26T06:13:49.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db2839f270 0x55db283b5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 0x55db283b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28378100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5e64200bd0 con 0x55db283b7bc0 2024-07-26T06:13:49.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5e641ffc00 con 0x55db2839f270 2024-07-26T06:13:49.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5e640b0140 con 0x55db283bf560 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e62a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 0x55db283b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e62a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 0x55db283b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36412/0 (socket says 172.21.15.163:36412) 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.194+0000 7f5e62a00640 1 -- 172.21.15.163:0/377407817 learned_addr learned my addr 172.21.15.163:0/377407817 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e63400640 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db2839f270 0x55db283b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e62a00640 1 -- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 msgr2=0x55db28378100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e63e00640 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28378100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e62a00640 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28378100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e62a00640 1 -- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db2839f270 msgr2=0x55db283b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e62a00640 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db2839f270 0x55db283b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e62a00640 1 -- 172.21.15.163:0/377407817 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2828fa20 con 0x55db283b7bc0 2024-07-26T06:13:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e62a00640 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 0x55db283b8da0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001800 tx=0x7f5e48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c1feba4d6e20fe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e4ea00640 1 -- 172.21.15.163:0/377407817 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50031040 con 0x55db283b7bc0 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e4ea00640 1 -- 172.21.15.163:0/377407817 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e500064b0 con 0x55db283b7bc0 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.195+0000 7f5e4ea00640 1 -- 172.21.15.163:0/377407817 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50003850 con 0x55db283b7bc0 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 -- 172.21.15.163:0/377407817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 msgr2=0x55db283b8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 0x55db283b8da0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5e50001800 tx=0x7f5e48005830 comp rx=0 tx=0).stop 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 -- 172.21.15.163:0/377407817 shutdown_connections 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283bf560 0x55db28378100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db283b7bc0 0x55db283b8da0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/377407817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db2839f270 0x55db283b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 -- 172.21.15.163:0/377407817 >> 172.21.15.163:0/377407817 conn(0x55db28379e50 msgr2=0x55db283bbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 -- 172.21.15.163:0/377407817 shutdown_connections 2024-07-26T06:13:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.196+0000 7f5e6cf88740 1 -- 172.21.15.163:0/377407817 wait complete. 2024-07-26T06:13:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.197+0000 7f5e6cf88740 1 -- start start 2024-07-26T06:13:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.197+0000 7f5e6cf88740 1 Processor -- start 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db2843acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283b7bc0 0x55db283c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283bf560 0x55db283c0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db283a2700 con 0x55db2839f270 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5e641ff5b0 con 0x55db283bf560 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55db28394670 con 0x55db283b7bc0 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e6cf88740 10 client.0 _ll_register_callbacks cb 0x7f5e48005740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db2843acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db2843acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36424/0 (socket says 172.21.15.163:36424) 2024-07-26T06:13:49.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 -- 172.21.15.163:0/1352006763 learned_addr learned my addr 172.21.15.163:0/1352006763 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 -- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283b7bc0 msgr2=0x55db283c09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283b7bc0 0x55db283c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 -- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283bf560 msgr2=0x55db283c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63e00640 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283bf560 0x55db283c0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283bf560 0x55db283c0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.198+0000 7f5e63400640 1 -- 172.21.15.163:0/1352006763 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db283df600 con 0x55db2839f270 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e63400640 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db2843acc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55db283b6490 tx=0x7f5e540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc5e072910f337e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:49.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e43e00640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e43e00640 1 -- 172.21.15.163:0/1352006763 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e54012ce0 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e43e00640 1 -- 172.21.15.163:0/1352006763 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5e54004220 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e500064b0 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e43e00640 1 -- 172.21.15.163:0/1352006763 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f5e54039050 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 10 client.4644 fetch_fsmap learned FSMap version 8 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55db28393db0 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e43e00640 1 -- 172.21.15.163:0/1352006763 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e500064b0 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e43e00640 1 -- 172.21.15.163:0/1352006763 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f5e5403c020 con 0x55db2839f270 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 10 client.4644 fetch_fsmap finished waiting for FSMap version 8 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 10 client.4644 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 20 client.4644 populate_metadata read hostname 'smithi163' 2024-07-26T06:13:49.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db28393720 0x55db28446a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 4 client.4644 mds_command: new command op to 4372 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.199+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 --> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] -- command(tid 0: {"prefix": "status"}) -- 0x55db283bb760 con 0x55db28393720 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e62a00640 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db28393720 0x55db28446a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e62a00640 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db28393720 0x55db28446a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f523fcec4a4703 server_cookie=bee92fcc02e005fc in_seq=0 out_seq=0 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e43e00640 10 client.4644 ms_handle_connect on v2:172.21.15.163:6834/2923632545 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e43e00640 1 -- 172.21.15.163:0/1352006763 <== mds.0 v2:172.21.15.163:6834/2923632545 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55db283bb760 con 0x55db28393720 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e43e00640 10 client.4644 handle_command_reply: tid=0 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 client.4644 shutdown 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 msgr2=0x55db2843acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db2843acc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55db283b6490 tx=0x7f5e540046c0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 shutdown_connections 2024-07-26T06:13:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:6834/2923632545,v1:172.21.15.163:6835/2923632545] conn(0x55db28393720 0x55db28446a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db283bf560 0x55db283c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55db283b7bc0 0x55db283c09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 --2- 172.21.15.163:0/1352006763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db2839f270 0x55db2843acc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 >> 172.21.15.163:0/1352006763 conn(0x55db28379e50 msgr2=0x55db283c01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.200+0000 7f5e43e00640 0 client.4644 ms_handle_reset on v2:172.21.15.163:6834/2923632545 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 shutdown_connections 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e6cf88740 1 -- 172.21.15.163:0/1352006763 wait complete. 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e6cf88740 20 client.4644 trim_cache size 0 max 16384 2024-07-26T06:13:49.207 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 4372, 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 14.174207967999999, 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 8, 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 21, 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 21, 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 20.242160908999999 2024-07-26T06:13:49.208 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:13:49.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5e44076300 msgr2=0x7f5e440787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5e44076300 0x7f5e440787c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009b20 tx=0x7f5e54005f90 comp rx=0 tx=0).stop 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 msgr2=0x7f5e641f4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 0x7f5e641f4080 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e48004780 tx=0x7f5e5c00ed20 comp rx=0 tx=0).stop 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 shutdown_connections 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5e44076300 0x7f5e440787c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5e641f3bf0 0x7f5e641f4080 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e64156690 0x7f5e6414f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 --2- 172.21.15.163:0/237533312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e48004800 0x7f5e64145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.201+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 >> 172.21.15.163:0/237533312 conn(0x7f5e640b0e30 msgr2=0x7f5e640ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:49.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.202+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 shutdown_connections 2024-07-26T06:13:49.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:49.202+0000 7f5e68c00640 1 -- 172.21.15.163:0/237533312 wait complete. 2024-07-26T06:13:49.219 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T06:13:49.229 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T06:13:49.229 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T06:13:49.229 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T06:13:49.229 INFO:teuthology.run_tasks:Running task kclient... 2024-07-26T06:13:49.254 INFO:tasks.kclient:Mounting kernel clients... 2024-07-26T06:13:49.255 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-07-26T06:13:49.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.255 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:13:49.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link delete ceph-brx 2024-07-26T06:13:49.393 INFO:teuthology.orchestra.run.smithi163.stderr:Cannot find device "ceph-brx" 2024-07-26T06:13:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:13:49.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.397 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:13:49.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link delete ceph-brx 2024-07-26T06:13:49.537 INFO:teuthology.orchestra.run.smithi163.stderr:Cannot find device "ceph-brx" 2024-07-26T06:13:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:13:49.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.541 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:13:49.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.600 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link delete ceph-brx 2024-07-26T06:13:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:Cannot find device "ceph-brx" 2024-07-26T06:13:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:13:49.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.696 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:13:49.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:49.746 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link delete ceph-brx 2024-07-26T06:13:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:Cannot find device "ceph-brx" 2024-07-26T06:13:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:13:49.834 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-07-26T06:13:49.834 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T06:13:49.835 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:13:49.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.048+0000 7f8033200640 1 -- 172.21.15.163:0/3012940308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80340cff20 msgr2=0x7f80340d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.048+0000 7f8033200640 1 --2- 172.21.15.163:0/3012940308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80340cff20 0x7f80340d03d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f80340747b0 tx=0x7f802c02a980 comp rx=0 tx=0).stop 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.048+0000 7f8033200640 1 -- 172.21.15.163:0/3012940308 shutdown_connections 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.048+0000 7f8033200640 1 --2- 172.21.15.163:0/3012940308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80340cff20 0x7f80340d03d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.048+0000 7f8033200640 1 --2- 172.21.15.163:0/3012940308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8034070a20 0x7f8034070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.048+0000 7f8033200640 1 --2- 172.21.15.163:0/3012940308 >> v2:172.21.15.164:3300/0 conn(0x7f803415b0c0 0x7f80340b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.049+0000 7f8033200640 1 -- 172.21.15.163:0/3012940308 >> 172.21.15.163:0/3012940308 conn(0x7f80340b3060 msgr2=0x7f80340b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.049+0000 7f8033200640 1 -- 172.21.15.163:0/3012940308 shutdown_connections 2024-07-26T06:13:50.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.049+0000 7f8033200640 1 -- 172.21.15.163:0/3012940308 wait complete. 2024-07-26T06:13:50.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.050+0000 7f8033200640 1 Processor -- start 2024-07-26T06:13:50.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.050+0000 7f8033200640 1 -- start start 2024-07-26T06:13:50.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.051+0000 7f8033200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 0x7f80340c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:50.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.051+0000 7f8033200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80340cff20 0x7f80340cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.051+0000 7f8033200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f803415b0c0 0x7f80340cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.051+0000 7f8033200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f80340b5b00 con 0x7f8034070a20 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.051+0000 7f8033200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f80340b5980 con 0x7f803415b0c0 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.051+0000 7f8033200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f80340b5c80 con 0x7f80340cff20 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 0x7f80340c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 0x7f80340c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36438/0 (socket says 172.21.15.163:36438) 2024-07-26T06:13:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 -- 172.21.15.163:0/2439326839 learned_addr learned my addr 172.21.15.163:0/2439326839 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:50.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8032800640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f803415b0c0 0x7f80340cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:50.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031400640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80340cff20 0x7f80340cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:50.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 -- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80340cff20 msgr2=0x7f80340cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80340cff20 0x7f80340cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 -- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f803415b0c0 msgr2=0x7f80340cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f803415b0c0 0x7f80340cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.052+0000 7f8031e00640 1 -- 172.21.15.163:0/2439326839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80340c9280 con 0x7f8034070a20 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.053+0000 7f8031e00640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 0x7f80340c8ba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f802400c970 tx=0x7f802400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.053+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024007bb0 con 0x7f8034070a20 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.053+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8024007e40 con 0x7f8034070a20 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.053+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024005280 con 0x7f8034070a20 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.053+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80340c9840 con 0x7f8034070a20 2024-07-26T06:13:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.053+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80340c9e00 con 0x7f8034070a20 2024-07-26T06:13:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.054+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80340d03d0 con 0x7f8034070a20 2024-07-26T06:13:50.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.056+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8024005420 con 0x7f8034070a20 2024-07-26T06:13:50.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.057+0000 7f802a000640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80140762d0 0x7f8014078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:50.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.057+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8024097b90 con 0x7f8034070a20 2024-07-26T06:13:50.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.058+0000 7f8031400640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80140762d0 0x7f8014078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:50.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.059+0000 7f8031400640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80140762d0 0x7f8014078790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80340b2520 tx=0x7f802000c460 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:50.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.061+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8024014030 con 0x7f8034070a20 2024-07-26T06:13:50.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.304+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f80340cbac0 con 0x7f8034070a20 2024-07-26T06:13:50.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.305+0000 7f802a000640 1 -- 172.21.15.163:0/2439326839 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f80240615a0 con 0x7f8034070a20 2024-07-26T06:13:50.306 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:13:50.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80140762d0 msgr2=0x7f8014078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:50.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80140762d0 0x7f8014078790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80340b2520 tx=0x7f802000c460 comp rx=0 tx=0).stop 2024-07-26T06:13:50.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 msgr2=0x7f80340c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 0x7f80340c8ba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f802400c970 tx=0x7f802400ce30 comp rx=0 tx=0).stop 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 shutdown_connections 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f80140762d0 0x7f8014078790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f803415b0c0 0x7f80340cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80340cff20 0x7f80340cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 --2- 172.21.15.163:0/2439326839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8034070a20 0x7f80340c8ba0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 >> 172.21.15.163:0/2439326839 conn(0x7f80340b3060 msgr2=0x7f80340b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.307+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 shutdown_connections 2024-07-26T06:13:50.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50.308+0000 7f8033200640 1 -- 172.21.15.163:0/2439326839 wait complete. 2024-07-26T06:13:50.326 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T06:13:50.326 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:13:50.326 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:13:50.326 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:13:50.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:13:50.327 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T06:13:50.327 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T06:13:50.327 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:13:50.327 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:13:50.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:50.327 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:13:50.376 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42574sec preferred_lft 42574sec 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:13:50.378 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:50.379 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link add name ceph-brx type bridge 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip addr flush dev ceph-brx 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set ceph-brx up 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T06:13:50.380 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:50.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:50.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:50.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:50.865 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:13:50.930 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:13:50.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:50.933 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:13:50.981 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:13:50.982 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:13:50.982 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:13:50.982 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:13:50.982 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:13:50.982 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:50.982 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:50.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:13:50.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:13:50.983 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:13:50.983 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:51.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:51.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:51.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:51.702 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:13:51.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:51.756 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list-id 2024-07-26T06:13:51.812 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:51.812 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:51.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:13:51.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T06:13:51.812 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:51.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:51.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:52.003 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T06:13:52.003 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:52.003 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:52.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T06:13:52.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T06:13:52.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T06:13:52.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T06:13:52.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T06:13:52.004 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:52.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:52.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:52.295 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:52.295 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:52.295 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:13:52.295 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T06:13:52.295 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:52.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:52.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:52.488 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:13:52.488 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:13:52.534 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:13:52.534 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:13:52.534 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:13:52.628 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:13:52.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:52.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:13:52.765 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:13:52.765 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:13:52.765 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:13:53.520 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:13:53.520 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:13:53.520 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:13:53.520 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:13:53.521 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:13:53.562 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:13:53.562 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:13:53.562 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:13:53.562 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:13:53.562 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:13:53.562 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:13:53.563 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:13:53.563 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:13:53.568 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:13:53.568 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:13:53.568 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:13:53.569 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:13:53.569 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:13:53.574 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:13:53.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:53.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:13:53.662 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:13:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:13:53.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:53.797 INFO:teuthology.orchestra.run.smithi163.stdout:client4606 2024-07-26T06:13:53.797 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:53.798 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:13:53.849 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:13:53.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:13:53.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:53.977 INFO:teuthology.orchestra.run.smithi163.stdout:client4606 2024-07-26T06:13:53.977 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:13:54.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:13:54.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:54.087 INFO:teuthology.orchestra.run.smithi163.stdout:client4606 2024-07-26T06:13:54.088 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1639b17-d73f-4cae-9471-025ba01f9572.client4606/status') 2024-07-26T06:13:54.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:54.224 INFO:teuthology.orchestra.run.smithi163.stdout:instance: client.4606 (3)172.21.15.163:0/970708705 2024-07-26T06:13:54.225 INFO:teuthology.orchestra.run.smithi163.stdout:blocklisted: false 2024-07-26T06:13:54.225 INFO:teuthology.orchestra.run.smithi163.stderr:0+1 records in 2024-07-26T06:13:54.225 INFO:teuthology.orchestra.run.smithi163.stderr:0+1 records out 2024-07-26T06:13:54.225 INFO:teuthology.orchestra.run.smithi163.stderr:70 bytes copied, 7.7936e-05 s, 898 kB/s 2024-07-26T06:13:54.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:54.237 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-07-26T06:13:54.237 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T06:13:54.238 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-07-26T06:13:54.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:13:54.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 -- 172.21.15.163:0/1206888302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0b2630 msgr2=0x7f7c2c0d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:54.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 --2- 172.21.15.163:0/1206888302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c0d2490 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c14005fd0 tx=0x7f7c1402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 -- 172.21.15.163:0/1206888302 shutdown_connections 2024-07-26T06:13:54.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 --2- 172.21.15.163:0/1206888302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c0d2490 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 -- 172.21.15.163:0/1206888302 >> 172.21.15.163:0/1206888302 conn(0x7f7c2c073a90 msgr2=0x7f7c2c073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:54.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 -- 172.21.15.163:0/1206888302 shutdown_connections 2024-07-26T06:13:54.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.441+0000 7f7c2be00640 1 -- 172.21.15.163:0/1206888302 wait complete. 2024-07-26T06:13:54.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.443+0000 7f7c2be00640 1 Processor -- start 2024-07-26T06:13:54.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.443+0000 7f7c2be00640 1 -- start start 2024-07-26T06:13:54.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:54.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c2c0b4d40 0x7f7c2c0caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:54.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 0x7f7c2c157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:54.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c15d9a0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b6fb0 con 0x7f7c2c0b2630 2024-07-26T06:13:54.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b2070 con 0x7f7c2c0b4d40 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 0x7f7c2c157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 0x7f7c2c157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36444/0 (socket says 172.21.15.163:36444) 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 -- 172.21.15.163:0/3715304219 learned_addr learned my addr 172.21.15.163:0/3715304219 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2aa00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2a000640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c2c0b4d40 0x7f7c2c0caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 -- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c2c0b4d40 msgr2=0x7f7c2c0caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c2c0b4d40 0x7f7c2c0caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 -- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2c0b2630 msgr2=0x7f7c2c155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2b400640 1 -- 172.21.15.163:0/3715304219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c159550 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.444+0000 7f7c2aa00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.445+0000 7f7c2b400640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 0x7f7c2c157b60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c3f0 tx=0x7f7c2400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.445+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24017070 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.445+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c240053c0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.445+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24003850 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.445+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2c159830 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.445+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2c156ed0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.446+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2c0d2490 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.448+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7c2401a070 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.448+0000 7f7c22a00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c08076300 0x7f7c080787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:54.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.449+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7c24096aa0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.449+0000 7f7c2aa00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c08076300 0x7f7c080787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:54.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.450+0000 7f7c2aa00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c08076300 0x7f7c080787c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0c68f0 tx=0x7f7c1c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:54.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.453+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c240604b0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.702+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7c2c0698f0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.703+0000 7f7c22a00640 1 -- 172.21.15.163:0/3715304219 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f7c2405fdb0 con 0x7f7c2c0d29d0 2024-07-26T06:13:54.703 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:13:54.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c08076300 msgr2=0x7f7c080787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:54.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c08076300 0x7f7c080787c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0c68f0 tx=0x7f7c1c009040 comp rx=0 tx=0).stop 2024-07-26T06:13:54.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 msgr2=0x7f7c2c157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:54.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 0x7f7c2c157b60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c3f0 tx=0x7f7c2400c8b0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 shutdown_connections 2024-07-26T06:13:54.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c08076300 0x7f7c080787c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c2c0d29d0 0x7f7c2c157b60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c2c0b4d40 0x7f7c2c0caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 --2- 172.21.15.163:0/3715304219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2c0b2630 0x7f7c2c155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 >> 172.21.15.163:0/3715304219 conn(0x7f7c2c073a90 msgr2=0x7f7c2c15e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 shutdown_connections 2024-07-26T06:13:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:54.706+0000 7f7c2be00640 1 -- 172.21.15.163:0/3715304219 wait complete. 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:13:54.725 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-07-26T06:13:54.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:54.726 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:13:54.776 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:13:54.776 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:54.777 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42570sec preferred_lft 42570sec 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:54.778 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:13:54.779 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:13:54.779 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:13:54.779 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:54.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:54.779 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:13:54.849 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:13:54.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:54.852 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:13:54.902 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:13:54.902 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:13:54.902 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:13:54.902 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:13:54.903 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:13:54.903 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:54.903 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:54.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:13:54.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:13:54.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:13:54.903 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:55.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:55.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:55.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:55.106 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:13:55.152 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:13:55.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:55.153 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list-id 2024-07-26T06:13:55.199 INFO:teuthology.orchestra.run.smithi163.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-07-26T06:13:55.200 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:55.200 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:55.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:13:55.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-07-26T06:13:55.200 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:55.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:55.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:55.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:55.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-07-26T06:13:55.451 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:13:55.451 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:13:55.451 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:13:55.451 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.451 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether a6:5a:32:fa:31:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::a45a:32ff:fefa:3128/64 scope link 2024-07-26T06:13:55.452 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:55.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.540 INFO:teuthology.orchestra.run.smithi163.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:13:55.541 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether a6:5a:32:fa:31:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-07-26T06:13:55.541 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-07-26T06:13:55.541 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.541 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::a45a:32ff:fefa:3128/64 scope link 2024-07-26T06:13:55.541 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:55.545 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-07-26T06:13:55.545 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:55.545 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:55.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-07-26T06:13:55.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-07-26T06:13:55.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-07-26T06:13:55.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-07-26T06:13:55.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-07-26T06:13:55.546 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:55.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:55.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:13:55.740 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:13:55.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:55.811 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:55.811 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:13:55.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.1 up 2024-07-26T06:13:55.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set dev brx.1 master ceph-brx 2024-07-26T06:13:55.811 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:13:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:56.017 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:13:56.017 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-07-26T06:13:56.062 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-07-26T06:13:56.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:13:56.063 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-07-26T06:13:56.157 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:13:56.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:56.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-07-26T06:13:56.287 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-07-26T06:13:56.287 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-07-26T06:13:56.287 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:13:56.307 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-07-26T06:13:56.307 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-07-26T06:13:56.307 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:13:56.307 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:13:56.308 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:13:56.332 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-07-26T06:13:56.332 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-07-26T06:13:56.332 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:13:56.332 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:13:56.333 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:13:56.333 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:13:56.333 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:13:56.333 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:13:56.334 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-07-26T06:13:56.334 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:13:56.334 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:13:56.335 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:13:56.335 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:13:56.338 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:13:56.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:56.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-07-26T06:13:56.407 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:13:56.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:13:56.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:56.520 INFO:teuthology.orchestra.run.smithi163.stdout:client4609 2024-07-26T06:13:56.520 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:13:56.521 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:13:56.565 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:13:56.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:13:56.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:56.719 INFO:teuthology.orchestra.run.smithi163.stdout:client4609 2024-07-26T06:13:56.720 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:13:56.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:13:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:56.823 INFO:teuthology.orchestra.run.smithi163.stdout:client4609 2024-07-26T06:13:56.823 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1639b17-d73f-4cae-9471-025ba01f9572.client4609/status') 2024-07-26T06:13:56.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:56.941 INFO:teuthology.orchestra.run.smithi163.stdout:instance: client.4609 (3)172.21.15.163:0/1145151226 2024-07-26T06:13:56.941 INFO:teuthology.orchestra.run.smithi163.stdout:blocklisted: false 2024-07-26T06:13:56.942 INFO:teuthology.orchestra.run.smithi163.stderr:0+1 records in 2024-07-26T06:13:56.942 INFO:teuthology.orchestra.run.smithi163.stderr:0+1 records out 2024-07-26T06:13:56.942 INFO:teuthology.orchestra.run.smithi163.stderr:71 bytes copied, 6.9353e-05 s, 1.0 MB/s 2024-07-26T06:13:56.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:56.954 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-07-26T06:13:56.954 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T06:13:56.954 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:13:56.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:13:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 -- 172.21.15.163:0/4130788703 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac0b23d0 msgr2=0x7facac0b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 --2- 172.21.15.163:0/4130788703 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac0b23d0 0x7facac0b6a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faca4005fd0 tx=0x7faca402aa10 comp rx=0 tx=0).stop 2024-07-26T06:13:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 -- 172.21.15.163:0/4130788703 shutdown_connections 2024-07-26T06:13:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 --2- 172.21.15.163:0/4130788703 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac0b23d0 0x7facac0b6a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 --2- 172.21.15.163:0/4130788703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facac0b1e80 0x7facac0b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 -- 172.21.15.163:0/4130788703 >> 172.21.15.163:0/4130788703 conn(0x7facac0cfda0 msgr2=0x7facac0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.143+0000 7facb0a00640 1 -- 172.21.15.163:0/4130788703 shutdown_connections 2024-07-26T06:13:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.144+0000 7facb0a00640 1 -- 172.21.15.163:0/4130788703 wait complete. 2024-07-26T06:13:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.145+0000 7facb0a00640 1 Processor -- start 2024-07-26T06:13:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.145+0000 7facb0a00640 1 -- start start 2024-07-26T06:13:57.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.146+0000 7facb0a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facac0b1e80 0x7facac157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:57.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.146+0000 7facb0a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facac0b23d0 0x7facac0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:57.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.146+0000 7facb0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 0x7facac0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:57.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.146+0000 7facb0a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7facac0b4f20 con 0x7facac15d5a0 2024-07-26T06:13:57.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.146+0000 7facb0a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7facac0b4da0 con 0x7facac0b23d0 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.146+0000 7facb0a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7facac0b50a0 con 0x7facac0b1e80 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 0x7facac0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 0x7facac0c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45818/0 (socket says 172.21.15.163:45818) 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 -- 172.21.15.163:0/1777036558 learned_addr learned my addr 172.21.15.163:0/1777036558 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facaaa00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facac0b23d0 0x7facac0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facab400640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facac0b1e80 0x7facac157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 -- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facac0b1e80 msgr2=0x7facac157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facac0b1e80 0x7facac157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 -- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facac0b23d0 msgr2=0x7facac0c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facac0b23d0 0x7facac0c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.147+0000 7facabe00640 1 -- 172.21.15.163:0/1777036558 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac153fb0 con 0x7facac15d5a0 2024-07-26T06:13:57.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.148+0000 7facabe00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 0x7facac0c7e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faca4006010 tx=0x7faca4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:57.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.148+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4047070 con 0x7facac15d5a0 2024-07-26T06:13:57.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.148+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faca4004250 con 0x7facac15d5a0 2024-07-26T06:13:57.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.148+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facac0c89d0 con 0x7facac15d5a0 2024-07-26T06:13:57.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.148+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4033050 con 0x7facac15d5a0 2024-07-26T06:13:57.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.148+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facac156b90 con 0x7facac15d5a0 2024-07-26T06:13:57.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.150+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facac0b5590 con 0x7facac15d5a0 2024-07-26T06:13:57.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.150+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faca404a070 con 0x7facac15d5a0 2024-07-26T06:13:57.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.151+0000 7faca3400640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fac8c0762d0 0x7fac8c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:13:57.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.152+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faca40b8940 con 0x7facac15d5a0 2024-07-26T06:13:57.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.152+0000 7facab400640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fac8c0762d0 0x7fac8c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:13:57.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.153+0000 7facab400640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fac8c0762d0 0x7fac8c078790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0015e0 tx=0x7fac9c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:13:57.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.156+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faca40822a0 con 0x7facac15d5a0 2024-07-26T06:13:57.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.407+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7facac069880 con 0x7facac15d5a0 2024-07-26T06:13:57.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.408+0000 7faca3400640 1 -- 172.21.15.163:0/1777036558 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7faca4081ba0 con 0x7facac15d5a0 2024-07-26T06:13:57.408 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:13:57.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fac8c0762d0 msgr2=0x7fac8c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:57.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fac8c0762d0 0x7fac8c078790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0015e0 tx=0x7fac9c009040 comp rx=0 tx=0).stop 2024-07-26T06:13:57.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 msgr2=0x7facac0c7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:13:57.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 0x7facac0c7e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faca4006010 tx=0x7faca4003f10 comp rx=0 tx=0).stop 2024-07-26T06:13:57.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 shutdown_connections 2024-07-26T06:13:57.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fac8c0762d0 0x7fac8c078790 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facac15d5a0 0x7facac0c7e60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facac0b23d0 0x7facac0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 --2- 172.21.15.163:0/1777036558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facac0b1e80 0x7facac157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:13:57.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 >> 172.21.15.163:0/1777036558 conn(0x7facac0cfda0 msgr2=0x7facac0d20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:13:57.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 shutdown_connections 2024-07-26T06:13:57.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:13:57.411+0000 7facb0a00640 1 -- 172.21.15.163:0/1777036558 wait complete. 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:13:57.430 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:13:57.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:57.431 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:13:57.488 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:13:57.488 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:13:57.488 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:13:57.489 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:13:57.490 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:13:57.490 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42545sec preferred_lft 42545sec 2024-07-26T06:13:57.490 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:13:57.490 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:13:57.490 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T06:13:57.490 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:57.490 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:13:57.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link add name ceph-brx type bridge 2024-07-26T06:13:57.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip addr flush dev ceph-brx 2024-07-26T06:13:57.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set ceph-brx up 2024-07-26T06:13:57.491 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T06:13:57.491 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:13:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:57.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:57.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:57.967 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:13:58.042 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:13:58.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:58.045 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:13:58.098 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:13:58.099 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:13:58.099 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:13:58.099 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:13:58.099 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:13:58.099 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:58.099 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:13:58.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:13:58.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:13:58.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:13:58.100 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:13:58.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:58.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:58.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:58.848 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:13:58.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:58.899 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list-id 2024-07-26T06:13:58.950 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:58.950 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:13:58.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:13:58.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-07-26T06:13:58.950 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:13:59.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:59.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:59.137 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-07-26T06:13:59.137 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:59.137 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:13:59.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-07-26T06:13:59.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T06:13:59.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-07-26T06:13:59.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-07-26T06:13:59.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-07-26T06:13:59.138 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:13:59.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:59.422 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:13:59.422 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:13:59.422 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:13:59.422 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T06:13:59.422 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:13:59.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:13:59.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:13:59.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:13:59.619 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:13:59.664 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:13:59.664 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:13:59.665 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:13:59.756 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:13:59.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:13:59.757 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:13:59.895 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:13:59.895 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:13:59.895 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:00.625 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:14:00.626 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:14:00.626 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:00.626 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:00.626 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:00.665 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:14:00.666 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:14:00.671 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:14:00.671 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:14:00.671 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:14:00.671 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:14:00.671 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:14:00.676 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:14:00.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:00.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:14:00.754 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:00.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:00.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:client4668 2024-07-26T06:14:00.873 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:00.873 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:00.923 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:01.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:01.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:01.088 INFO:teuthology.orchestra.run.smithi164.stdout:client4668 2024-07-26T06:14:01.088 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:01.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:01.266 INFO:teuthology.orchestra.run.smithi164.stdout:client4668 2024-07-26T06:14:01.267 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1639b17-d73f-4cae-9471-025ba01f9572.client4668/status') 2024-07-26T06:14:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:01.405 INFO:teuthology.orchestra.run.smithi164.stdout:instance: client.4668 (3)172.21.15.164:0/381908860 2024-07-26T06:14:01.406 INFO:teuthology.orchestra.run.smithi164.stdout:blocklisted: false 2024-07-26T06:14:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2024-07-26T06:14:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2024-07-26T06:14:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:70 bytes copied, 0.000103213 s, 678 kB/s 2024-07-26T06:14:01.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:01.419 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-07-26T06:14:01.419 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T06:14:01.419 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-07-26T06:14:01.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:14:01.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 -- 172.21.15.163:0/695438258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cbad0 msgr2=0x7f00d40cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:01.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 --2- 172.21.15.163:0/695438258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cbad0 0x7f00d40cd530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f00c800a940 tx=0x7f00c802b850 comp rx=0 tx=0).stop 2024-07-26T06:14:01.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 -- 172.21.15.163:0/695438258 shutdown_connections 2024-07-26T06:14:01.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 --2- 172.21.15.163:0/695438258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40cd000 0x7f00d40cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 --2- 172.21.15.163:0/695438258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cbad0 0x7f00d40cd530 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 -- 172.21.15.163:0/695438258 >> 172.21.15.163:0/695438258 conn(0x7f00d40b0e30 msgr2=0x7f00d40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:01.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 -- 172.21.15.163:0/695438258 shutdown_connections 2024-07-26T06:14:01.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.638+0000 7f00d9e00640 1 -- 172.21.15.163:0/695438258 wait complete. 2024-07-26T06:14:01.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.639+0000 7f00d9e00640 1 Processor -- start 2024-07-26T06:14:01.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.640+0000 7f00d9e00640 1 -- start start 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40b4420 0x7f00d4157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 0x7f00d41467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00d4146d30 0x7f00d41e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f00d41594f0 con 0x7f00d40cd000 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4159370 con 0x7f00d4146d30 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f00d4159670 con 0x7f00d40b4420 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 0x7f00d41467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:01.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 0x7f00d41467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45844/0 (socket says 172.21.15.163:45844) 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 -- 172.21.15.163:0/1007622338 learned_addr learned my addr 172.21.15.163:0/1007622338 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d8a00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40b4420 0x7f00d4157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d9400640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00d4146d30 0x7f00d41e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 -- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40b4420 msgr2=0x7f00d4157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40b4420 0x7f00d4157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 -- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00d4146d30 msgr2=0x7f00d41e7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00d4146d30 0x7f00d41e7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d3e00640 1 -- 172.21.15.163:0/1007622338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d41e7fc0 con 0x7f00d40cd000 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.641+0000 7f00d8a00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40b4420 0x7f00d4157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.642+0000 7f00d3e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 0x7f00d41467f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c450 tx=0x7f00c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:01.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.642+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0016020 con 0x7f00d40cd000 2024-07-26T06:14:01.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.642+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00c0012920 con 0x7f00d40cd000 2024-07-26T06:14:01.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.642+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d41e82a0 con 0x7f00d40cd000 2024-07-26T06:14:01.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.642+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0012d70 con 0x7f00d40cd000 2024-07-26T06:14:01.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.642+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d41e8860 con 0x7f00d40cd000 2024-07-26T06:14:01.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.643+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d4145350 con 0x7f00d40cd000 2024-07-26T06:14:01.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.644+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00c0019020 con 0x7f00d40cd000 2024-07-26T06:14:01.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.645+0000 7f00d0c00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00b40762d0 0x7f00b4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:01.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.645+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f00c0097850 con 0x7f00d40cd000 2024-07-26T06:14:01.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.646+0000 7f00d8a00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00b40762d0 0x7f00b4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:01.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.648+0000 7f00d8a00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00b40762d0 0x7f00b4078790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f00c8036000 tx=0x7f00c8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:01.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.650+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f00c0020020 con 0x7f00d40cd000 2024-07-26T06:14:01.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.898+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f00d40add00 con 0x7f00d40cd000 2024-07-26T06:14:01.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.899+0000 7f00d0c00640 1 -- 172.21.15.163:0/1007622338 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f00c0061110 con 0x7f00d40cd000 2024-07-26T06:14:01.900 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00b40762d0 msgr2=0x7f00b4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00b40762d0 0x7f00b4078790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f00c8036000 tx=0x7f00c8004730 comp rx=0 tx=0).stop 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 msgr2=0x7f00d41467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 0x7f00d41467f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c450 tx=0x7f00c000c910 comp rx=0 tx=0).stop 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 shutdown_connections 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f00b40762d0 0x7f00b4078790 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00d4146d30 0x7f00d41e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f00d40cd000 0x7f00d41467f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 --2- 172.21.15.163:0/1007622338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00d40b4420 0x7f00d4157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:01.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.901+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 >> 172.21.15.163:0/1007622338 conn(0x7f00d40b0e30 msgr2=0x7f00d40b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:01.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.902+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 shutdown_connections 2024-07-26T06:14:01.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:01.902+0000 7f00d9e00640 1 -- 172.21.15.163:0/1007622338 wait complete. 2024-07-26T06:14:01.921 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T06:14:01.921 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:14:01.921 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:14:01.922 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-07-26T06:14:01.922 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-07-26T06:14:01.922 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T06:14:01.922 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T06:14:01.922 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:14:01.922 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-07-26T06:14:01.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:01.923 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:14:01.980 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:14:01.980 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:14:01.980 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:14:01.980 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:01.980 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:14:01.980 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42541sec preferred_lft 42541sec 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:14:01.981 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:14:01.982 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:01.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:01.983 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:14:02.062 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:14:02.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:02.066 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:14:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:14:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:14:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:14:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:14:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:14:02.119 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:14:02.120 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:14:02.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:14:02.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:14:02.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:14:02.120 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:14:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:02.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:02.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:02.328 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:14:02.376 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:14:02.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:02.377 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list-id 2024-07-26T06:14:02.424 INFO:teuthology.orchestra.run.smithi164.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-07-26T06:14:02.425 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:14:02.425 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:14:02.426 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:14:02.426 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-07-26T06:14:02.426 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:14:02.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:02.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:02.656 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-07-26T06:14:02.731 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:14:02.731 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:14:02.731 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:14:02.731 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 72:1c:2c:bc:4e:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::701c:2cff:febc:4edc/64 scope link 2024-07-26T06:14:02.732 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:02.735 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-07-26T06:14:02.816 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:14:02.816 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:14:02.816 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 72:1c:2c:bc:4e:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::701c:2cff:febc:4edc/64 scope link 2024-07-26T06:14:02.817 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:02.820 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-07-26T06:14:02.821 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:14:02.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:03.103 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:14:03.103 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:14:03.104 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.1 up 2024-07-26T06:14:03.104 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set dev brx.1 master ceph-brx 2024-07-26T06:14:03.104 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:14:03.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:03.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:03.313 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:14:03.313 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-07-26T06:14:03.359 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-07-26T06:14:03.359 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:14:03.359 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-07-26T06:14:03.451 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:14:03.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:03.451 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-07-26T06:14:03.568 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-07-26T06:14:03.568 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-07-26T06:14:03.569 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:03.588 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-07-26T06:14:03.588 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-07-26T06:14:03.589 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:03.589 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:03.589 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:03.616 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:14:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:14:03.618 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-07-26T06:14:03.618 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:14:03.618 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:14:03.619 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:14:03.619 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:14:03.624 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:14:03.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:03.624 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-07-26T06:14:03.696 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:03.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:03.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:03.812 INFO:teuthology.orchestra.run.smithi164.stdout:client4493 2024-07-26T06:14:03.813 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:03.813 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:03.864 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:04.028 INFO:teuthology.orchestra.run.smithi164.stdout:client4493 2024-07-26T06:14:04.029 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:04.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:client4493 2024-07-26T06:14:04.148 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1639b17-d73f-4cae-9471-025ba01f9572.client4493/status') 2024-07-26T06:14:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:04.282 INFO:teuthology.orchestra.run.smithi164.stdout:instance: client.4493 (3)192.168.0.2:0/774190287 2024-07-26T06:14:04.282 INFO:teuthology.orchestra.run.smithi164.stdout:blocklisted: false 2024-07-26T06:14:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2024-07-26T06:14:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2024-07-26T06:14:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:68 bytes copied, 7.7906e-05 s, 873 kB/s 2024-07-26T06:14:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:04.296 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-07-26T06:14:04.399 INFO:tasks.cephfs_test_runner:Starting test: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate binary .meta file on legacy subvol root doesn't break the system 2024-07-26T06:14:04.399 INFO:tasks.cephfs_test_runner:test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:14:04.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:04.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 -- 172.21.15.163:0/3516886534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa940b4420 msgr2=0x7ffa940cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:04.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3516886534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa940b4420 0x7ffa940cda70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffa940b1e30 tx=0x7ffa7c02a530 comp rx=0 tx=0).stop 2024-07-26T06:14:04.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 -- 172.21.15.163:0/3516886534 shutdown_connections 2024-07-26T06:14:04.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3516886534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa940b4420 0x7ffa940cda70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3516886534 >> v2:172.21.15.164:3300/0 conn(0x7ffa940cbad0 0x7ffa940cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 -- 172.21.15.163:0/3516886534 >> 172.21.15.163:0/3516886534 conn(0x7ffa940b0e30 msgr2=0x7ffa940b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:04.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 -- 172.21.15.163:0/3516886534 shutdown_connections 2024-07-26T06:14:04.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.620+0000 7ffa99a00640 1 -- 172.21.15.163:0/3516886534 wait complete. 2024-07-26T06:14:04.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.622+0000 7ffa99a00640 1 Processor -- start 2024-07-26T06:14:04.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.622+0000 7ffa99a00640 1 -- start start 2024-07-26T06:14:04.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa940b4420 0x7ffa940ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:04.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa940cd000 0x7ffa94154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 0x7ffa941e5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa941594f0 con 0x7ffa94147070 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffa94159370 con 0x7ffa940b4420 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa94159670 con 0x7ffa940cd000 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 0x7ffa941e5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 0x7ffa941e5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45860/0 (socket says 172.21.15.163:45860) 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.623+0000 7ffa99000640 1 -- 172.21.15.163:0/3391262837 learned_addr learned my addr 172.21.15.163:0/3391262837 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa93e00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa940b4420 0x7ffa940ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99000640 1 -- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa940cd000 msgr2=0x7ffa94154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99000640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa940cd000 0x7ffa94154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99000640 1 -- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa940b4420 msgr2=0x7ffa940ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99000640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa940b4420 0x7ffa940ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99000640 1 -- 172.21.15.163:0/3391262837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa941585d0 con 0x7ffa94147070 2024-07-26T06:14:04.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99000640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 0x7ffa941e5550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c00cb10 tx=0x7ffa8c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:04.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c016020 con 0x7ffa94147070 2024-07-26T06:14:04.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa8c00ad00 con 0x7ffa94147070 2024-07-26T06:14:04.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c012680 con 0x7ffa94147070 2024-07-26T06:14:04.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.624+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa94156a00 con 0x7ffa94147070 2024-07-26T06:14:04.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.625+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa941588b0 con 0x7ffa94147070 2024-07-26T06:14:04.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.626+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa94156df0 con 0x7ffa94147070 2024-07-26T06:14:04.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.627+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa8c019020 con 0x7ffa94147070 2024-07-26T06:14:04.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.628+0000 7ffa90c00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa700762d0 0x7ffa70078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:04.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.628+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffa8c097c00 con 0x7ffa94147070 2024-07-26T06:14:04.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.629+0000 7ffa93e00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa700762d0 0x7ffa70078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:04.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.630+0000 7ffa93e00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa700762d0 0x7ffa70078790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005fd0 tx=0x7ffa84005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:04.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.633+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa8c020030 con 0x7ffa94147070 2024-07-26T06:14:04.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.840+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa94156bd0 con 0x7ffa94147070 2024-07-26T06:14:04.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.841+0000 7ffa90c00640 1 -- 172.21.15.163:0/3391262837 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7ffa8c061440 con 0x7ffa94147070 2024-07-26T06:14:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:04.843 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":21,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:36.040341+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:04.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.844+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa700762d0 msgr2=0x7ffa70078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:04.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.844+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa700762d0 0x7ffa70078790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005fd0 tx=0x7ffa84005850 comp rx=0 tx=0).stop 2024-07-26T06:14:04.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.844+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 msgr2=0x7ffa941e5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:04.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.844+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 0x7ffa941e5550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c00cb10 tx=0x7ffa8c00a060 comp rx=0 tx=0).stop 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 shutdown_connections 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa700762d0 0x7ffa70078790 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa94147070 0x7ffa941e5550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa940cd000 0x7ffa94154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 --2- 172.21.15.163:0/3391262837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa940b4420 0x7ffa940ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 >> 172.21.15.163:0/3391262837 conn(0x7ffa940b0e30 msgr2=0x7ffa940b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:04.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 shutdown_connections 2024-07-26T06:14:04.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:04.845+0000 7ffa99a00640 1 -- 172.21.15.163:0/3391262837 wait complete. 2024-07-26T06:14:04.865 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:13:21.677101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-07-26T06:13:32.708837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-07-26T06:13:33.488054+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:14:04.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f15a4c00640 1 -- 172.21.15.163:0/2266287620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584003fd0 msgr2=0x7f1584004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f15a4c00640 1 --2- 172.21.15.163:0/2266287620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584003fd0 0x7f1584004800 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f158c009b50 tx=0x7f1584005830 comp rx=0 tx=0).stop 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f159fe00640 1 -- 172.21.15.163:0/2266287620 reap_dead start 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f15a4c00640 1 -- 172.21.15.163:0/2266287620 shutdown_connections 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f15a4c00640 1 -- 172.21.15.163:0/2266287620 >> 172.21.15.163:0/2266287620 conn(0x7f15a00b0e30 msgr2=0x7f15a00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f15a4c00640 1 -- 172.21.15.163:0/2266287620 shutdown_connections 2024-07-26T06:14:05.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.071+0000 7f15a4c00640 1 -- 172.21.15.163:0/2266287620 wait complete. 2024-07-26T06:14:05.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.073+0000 7f15a4c00640 1 Processor -- start 2024-07-26T06:14:05.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.073+0000 7f15a4c00640 1 -- start start 2024-07-26T06:14:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.074+0000 7f15a4c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1584003fd0 0x7f15a01e8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.074+0000 7f15a4c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a0143410 0x7f15a0143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.074+0000 7f15a4c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 0x7f15a01e4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.074+0000 7f15a4c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15a00718c0 con 0x7f15a01e4820 2024-07-26T06:14:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.074+0000 7f15a4c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0071740 con 0x7f1584003fd0 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.074+0000 7f15a4c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f15a0071fe0 con 0x7f15a0143410 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 0x7f15a01e4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 0x7f15a01e4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33546/0 (socket says 172.21.15.163:33546) 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 -- 172.21.15.163:0/1590559127 learned_addr learned my addr 172.21.15.163:0/1590559127 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 -- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a0143410 msgr2=0x7f15a0143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159f400640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1584003fd0 0x7f15a01e8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a0143410 0x7f15a0143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 -- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1584003fd0 msgr2=0x7f15a01e8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1584003fd0 0x7f15a01e8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.075+0000 7f159fe00640 1 -- 172.21.15.163:0/1590559127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a01e6000 con 0x7f15a01e4820 2024-07-26T06:14:05.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.076+0000 7f159fe00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 0x7f15a01e4ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f15a00b1e30 tx=0x7f159800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:05.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.076+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598004b20 con 0x7f15a01e4820 2024-07-26T06:14:05.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.076+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1598004cc0 con 0x7f15a01e4820 2024-07-26T06:14:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.076+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598003bd0 con 0x7f15a01e4820 2024-07-26T06:14:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.076+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a01e62e0 con 0x7f15a01e4820 2024-07-26T06:14:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.076+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a01e68a0 con 0x7f15a01e4820 2024-07-26T06:14:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.077+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a00ced60 con 0x7f15a01e4820 2024-07-26T06:14:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.078+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1598004030 con 0x7f15a01e4820 2024-07-26T06:14:05.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.079+0000 7f1597400640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1580076300 0x7f15800787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.079+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f15980980f0 con 0x7f15a01e4820 2024-07-26T06:14:05.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.080+0000 7f159f400640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1580076300 0x7f15800787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.081+0000 7f159f400640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1580076300 0x7f15800787c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1590000fb0 tx=0x7f1590001e00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:05.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.084+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1598061a40 con 0x7f15a01e4820 2024-07-26T06:14:05.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.288+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15a01544f0 con 0x7f15a01e4820 2024-07-26T06:14:05.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.289+0000 7f1597400640 1 -- 172.21.15.163:0/1590559127 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f1598061340 con 0x7f15a01e4820 2024-07-26T06:14:05.290 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.291+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1580076300 msgr2=0x7f15800787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.291+0000 7f15a4c00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1580076300 0x7f15800787c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1590000fb0 tx=0x7f1590001e00 comp rx=0 tx=0).stop 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.291+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 msgr2=0x7f15a01e4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.291+0000 7f15a4c00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 0x7f15a01e4ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f15a00b1e30 tx=0x7f159800f3c0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 shutdown_connections 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1580076300 0x7f15800787c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15a01e4820 0x7f15a01e4ca0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a0143410 0x7f15a0143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 --2- 172.21.15.163:0/1590559127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1584003fd0 0x7f15a01e8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 >> 172.21.15.163:0/1590559127 conn(0x7f15a00b0e30 msgr2=0x7f15a0154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:05.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 shutdown_connections 2024-07-26T06:14:05.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.292+0000 7f15a4c00640 1 -- 172.21.15.163:0/1590559127 wait complete. 2024-07-26T06:14:05.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.509+0000 7f40ed200640 1 -- 172.21.15.163:0/1423319122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40cc004d40 msgr2=0x7f40cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.509+0000 7f40ed200640 1 --2- 172.21.15.163:0/1423319122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40cc004d40 0x7f40cc0051c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f40e0005fd0 tx=0x7f40cc005830 comp rx=0 tx=0).stop 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.509+0000 7f40e7e00640 1 -- 172.21.15.163:0/1423319122 reap_dead start 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.509+0000 7f40ed200640 1 -- 172.21.15.163:0/1423319122 shutdown_connections 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.509+0000 7f40ed200640 1 -- 172.21.15.163:0/1423319122 >> 172.21.15.163:0/1423319122 conn(0x7f40e80b0e30 msgr2=0x7f40e80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.509+0000 7f40ed200640 1 -- 172.21.15.163:0/1423319122 shutdown_connections 2024-07-26T06:14:05.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.510+0000 7f40ed200640 1 -- 172.21.15.163:0/1423319122 wait complete. 2024-07-26T06:14:05.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.511+0000 7f40ed200640 1 Processor -- start 2024-07-26T06:14:05.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.511+0000 7f40ed200640 1 -- start start 2024-07-26T06:14:05.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40ed200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40cc004d40 0x7f40e8149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40ed200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 0x7f40e814de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40ed200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e814e370 0x7f40e814a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40ed200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40e8164190 con 0x7f40e815cf40 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40ed200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40e80736c0 con 0x7f40e814e370 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40ed200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f40e8164310 con 0x7f40cc004d40 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.512+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 0x7f40e814de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 0x7f40e814de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33576/0 (socket says 172.21.15.163:33576) 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 -- 172.21.15.163:0/925741489 learned_addr learned my addr 172.21.15.163:0/925741489 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 -- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40cc004d40 msgr2=0x7f40e8149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40cc004d40 0x7f40e8149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 -- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e814e370 msgr2=0x7f40e814a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e814e370 0x7f40e814a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 -- 172.21.15.163:0/925741489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e81f5160 con 0x7f40e815cf40 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40dfe00640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 0x7f40e814de30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40d0007fd0 tx=0x7f40d000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0007030 con 0x7f40e815cf40 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40d00071d0 con 0x7f40e815cf40 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e81f4550 con 0x7f40e815cf40 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.513+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0007620 con 0x7f40e815cf40 2024-07-26T06:14:05.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.514+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e81f4b10 con 0x7f40e815cf40 2024-07-26T06:14:05.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.515+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e8071530 con 0x7f40e815cf40 2024-07-26T06:14:05.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.516+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f40d0011070 con 0x7f40e815cf40 2024-07-26T06:14:05.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.516+0000 7f40df400640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40c40762d0 0x7f40c4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.517+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f40d00975b0 con 0x7f40e815cf40 2024-07-26T06:14:05.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.517+0000 7f40e7400640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40c40762d0 0x7f40c4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.518+0000 7f40e7400640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40c40762d0 0x7f40c4078790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40e814f540 tx=0x7f40d8006920 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:05.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.520+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f40d0060e70 con 0x7f40e815cf40 2024-07-26T06:14:05.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.709+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f40e80b34f0 con 0x7f40e815cf40 2024-07-26T06:14:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.710+0000 7f40df400640 1 -- 172.21.15.163:0/925741489 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f40d0060770 con 0x7f40e815cf40 2024-07-26T06:14:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 32 2024-07-26T06:14:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40c40762d0 msgr2=0x7f40c4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40c40762d0 0x7f40c4078790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40e814f540 tx=0x7f40d8006920 comp rx=0 tx=0).stop 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 msgr2=0x7f40e814de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 0x7f40e814de30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40d0007fd0 tx=0x7f40d000ccd0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 shutdown_connections 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f40c40762d0 0x7f40c4078790 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40e814e370 0x7f40e814a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40e815cf40 0x7f40e814de30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 --2- 172.21.15.163:0/925741489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40cc004d40 0x7f40e8149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.712+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 >> 172.21.15.163:0/925741489 conn(0x7f40e80b0e30 msgr2=0x7f40e80af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.713+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 shutdown_connections 2024-07-26T06:14:05.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.713+0000 7f40ed200640 1 -- 172.21.15.163:0/925741489 wait complete. 2024-07-26T06:14:05.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-07-26T06:14:05.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 -- 172.21.15.163:0/2767517739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc0cff20 msgr2=0x7f6fcc0d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2767517739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc0cff20 0x7f6fcc0d03d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0747b0 tx=0x7f6fc402a980 comp rx=0 tx=0).stop 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 -- 172.21.15.163:0/2767517739 shutdown_connections 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2767517739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc0cff20 0x7f6fcc0d03d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2767517739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fcc070a20 0x7f6fcc070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2767517739 >> v2:172.21.15.164:3300/0 conn(0x7f6fcc15b0c0 0x7f6fcc0b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 -- 172.21.15.163:0/2767517739 >> 172.21.15.163:0/2767517739 conn(0x7f6fcc0b3060 msgr2=0x7f6fcc0b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 -- 172.21.15.163:0/2767517739 shutdown_connections 2024-07-26T06:14:05.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.927+0000 7f6fd2000640 1 -- 172.21.15.163:0/2767517739 wait complete. 2024-07-26T06:14:05.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.929+0000 7f6fd2000640 1 Processor -- start 2024-07-26T06:14:05.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.929+0000 7f6fd2000640 1 -- start start 2024-07-26T06:14:05.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd2000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 0x7f6fcc0b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd2000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fcc0cff20 0x7f6fcc1f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd2000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fcc15b0c0 0x7f6fcc0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd2000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc0b5b00 con 0x7f6fcc070a20 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd2000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc0b5980 con 0x7f6fcc15b0c0 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd2000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc0b5c80 con 0x7f6fcc0cff20 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd0c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 0x7f6fcc0b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd0c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 0x7f6fcc0b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33608/0 (socket says 172.21.15.163:33608) 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.930+0000 7f6fd0c00640 1 -- 172.21.15.163:0/2843124348 learned_addr learned my addr 172.21.15.163:0/2843124348 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd0c00640 1 -- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fcc0cff20 msgr2=0x7f6fcc1f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd0c00640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fcc0cff20 0x7f6fcc1f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd0c00640 1 -- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fcc15b0c0 msgr2=0x7f6fcc0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:05.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd0c00640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fcc15b0c0 0x7f6fcc0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:05.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd0c00640 1 -- 172.21.15.163:0/2843124348 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc1f5250 con 0x7f6fcc070a20 2024-07-26T06:14:05.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd0c00640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 0x7f6fcc0b0170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d740 tx=0x7f6fbc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:05.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017070 con 0x7f6fcc070a20 2024-07-26T06:14:05.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fbc009cc0 con 0x7f6fcc070a20 2024-07-26T06:14:05.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc0a94d0 con 0x7f6fcc070a20 2024-07-26T06:14:05.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0106c0 con 0x7f6fcc070a20 2024-07-26T06:14:05.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.931+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc1f54f0 con 0x7f6fcc070a20 2024-07-26T06:14:05.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.932+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fcc0d03d0 con 0x7f6fcc070a20 2024-07-26T06:14:05.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.933+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6fbc01a070 con 0x7f6fcc070a20 2024-07-26T06:14:05.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.934+0000 7f6fc9600640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6fa80762d0 0x7f6fa8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:05.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.935+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6fbc0978c0 con 0x7f6fcc070a20 2024-07-26T06:14:05.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.935+0000 7f6fc3e00640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6fa80762d0 0x7f6fa8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:05.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.936+0000 7f6fc3e00640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6fa80762d0 0x7f6fa8078790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4005d80 tx=0x7f6fb4005cb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:05.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:05.938+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6fbc061210 con 0x7f6fcc070a20 2024-07-26T06:14:06.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.144+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6fcc156e30 con 0x7f6fcc070a20 2024-07-26T06:14:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.145+0000 7f6fc9600640 1 -- 172.21.15.163:0/2843124348 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f6fbc069030 con 0x7f6fcc070a20 2024-07-26T06:14:06.146 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 64 2024-07-26T06:14:06.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6fa80762d0 msgr2=0x7f6fa8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:06.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6fa80762d0 0x7f6fa8078790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4005d80 tx=0x7f6fb4005cb0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 msgr2=0x7f6fcc0b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:06.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 0x7f6fcc0b0170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d740 tx=0x7f6fbc00dc00 comp rx=0 tx=0).stop 2024-07-26T06:14:06.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 shutdown_connections 2024-07-26T06:14:06.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6fa80762d0 0x7f6fa8078790 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fcc15b0c0 0x7f6fcc0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fcc0cff20 0x7f6fcc1f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 --2- 172.21.15.163:0/2843124348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fcc070a20 0x7f6fcc0b0170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 >> 172.21.15.163:0/2843124348 conn(0x7f6fcc0b3060 msgr2=0x7f6fcc15ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 shutdown_connections 2024-07-26T06:14:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.148+0000 7f6fd2000640 1 -- 172.21.15.163:0/2843124348 wait complete. 2024-07-26T06:14:06.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.368+0000 7fa21ee00640 1 -- 172.21.15.163:0/3911882106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 msgr2=0x7fa2180cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.368+0000 7fa21ee00640 1 --2- 172.21.15.163:0/3911882106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 0x7fa2180cd0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa208005fd0 tx=0x7fa20802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.368+0000 7fa21ee00640 1 -- 172.21.15.163:0/3911882106 shutdown_connections 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.368+0000 7fa21ee00640 1 --2- 172.21.15.163:0/3911882106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2180cc1c0 0x7fa2180b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.368+0000 7fa21ee00640 1 --2- 172.21.15.163:0/3911882106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 0x7fa2180cd0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.368+0000 7fa21ee00640 1 -- 172.21.15.163:0/3911882106 >> 172.21.15.163:0/3911882106 conn(0x7fa2180b0e30 msgr2=0x7fa2180b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.369+0000 7fa21ee00640 1 -- 172.21.15.163:0/3911882106 shutdown_connections 2024-07-26T06:14:06.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.369+0000 7fa21ee00640 1 -- 172.21.15.163:0/3911882106 wait complete. 2024-07-26T06:14:06.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.370+0000 7fa21ee00640 1 Processor -- start 2024-07-26T06:14:06.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.371+0000 7fa21ee00640 1 -- start start 2024-07-26T06:14:06.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21ee00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 0x7fa2181f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21ee00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2180cc1c0 0x7fa2180ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21ee00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 0x7fa2180ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21ee00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2180b25c0 con 0x7fa21815a6b0 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21ee00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa2180b2440 con 0x7fa2180741f0 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21ee00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2180b2740 con 0x7fa2180cc1c0 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21e400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 0x7fa2180ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21e400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 0x7fa2180ae7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33614/0 (socket says 172.21.15.163:33614) 2024-07-26T06:14:06.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21e400640 1 -- 172.21.15.163:0/151813868 learned_addr learned my addr 172.21.15.163:0/151813868 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:06.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21da00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 0x7fa2181f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.372+0000 7fa21d000640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2180cc1c0 0x7fa2180ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21e400640 1 -- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2180cc1c0 msgr2=0x7fa2180ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21e400640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2180cc1c0 0x7fa2180ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21e400640 1 -- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 msgr2=0x7fa2181f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21e400640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 0x7fa2181f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21e400640 1 -- 172.21.15.163:0/151813868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa218156070 con 0x7fa21815a6b0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21e400640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 0x7fa2180ae7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa21000eb40 tx=0x7fa21000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa210004b20 con 0x7fa21815a6b0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa210004ce0 con 0x7fa21815a6b0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa210005280 con 0x7fa21815a6b0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa218156350 con 0x7fa21815a6b0 2024-07-26T06:14:06.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.373+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21814e2c0 con 0x7fa21815a6b0 2024-07-26T06:14:06.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.375+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2180cd0c0 con 0x7fa21815a6b0 2024-07-26T06:14:06.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.376+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa210005420 con 0x7fa21815a6b0 2024-07-26T06:14:06.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.377+0000 7fa215600640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa1f80762d0 0x7fa1f8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:06.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.377+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa210098400 con 0x7fa21815a6b0 2024-07-26T06:14:06.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.377+0000 7fa21da00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa1f80762d0 0x7fa1f8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:06.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.379+0000 7fa21da00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa1f80762d0 0x7fa1f8078790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa208033000 tx=0x7fa208004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:06.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.381+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa210061cc0 con 0x7fa21815a6b0 2024-07-26T06:14:06.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:06.647+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fa2180b4aa0 con 0x7fa21815a6b0 2024-07-26T06:14:07.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.219+0000 7fa215600640 1 -- 172.21.15.163:0/151813868 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v50) ==== 161+0+0 (secure 0 0 0) 0x7fa2100615c0 con 0x7fa21815a6b0 2024-07-26T06:14:07.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa1f80762d0 msgr2=0x7fa1f8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa1f80762d0 0x7fa1f8078790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa208033000 tx=0x7fa208004730 comp rx=0 tx=0).stop 2024-07-26T06:14:07.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 msgr2=0x7fa2180ae7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 0x7fa2180ae7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa21000eb40 tx=0x7fa21000c2d0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 shutdown_connections 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa1f80762d0 0x7fa1f8078790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa21815a6b0 0x7fa2180ae7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2180cc1c0 0x7fa2180ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 --2- 172.21.15.163:0/151813868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2180741f0 0x7fa2181f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 >> 172.21.15.163:0/151813868 conn(0x7fa2180b0e30 msgr2=0x7fa21814fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.222+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 shutdown_connections 2024-07-26T06:14:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.223+0000 7fa21ee00640 1 -- 172.21.15.163:0/151813868 wait complete. 2024-07-26T06:14:07.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.435+0000 7fbe3ac00640 1 -- 172.21.15.163:0/3511551991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340cbad0 msgr2=0x7fbe340cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.435+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/3511551991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340cbad0 0x7fbe340cd530 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009830 tx=0x7fbe2802a510 comp rx=0 tx=0).stop 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.436+0000 7fbe3ac00640 1 -- 172.21.15.163:0/3511551991 shutdown_connections 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.436+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/3511551991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340cbad0 0x7fbe340cd530 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.436+0000 7fbe3ac00640 1 -- 172.21.15.163:0/3511551991 >> 172.21.15.163:0/3511551991 conn(0x7fbe340b0e30 msgr2=0x7fbe340b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.436+0000 7fbe3ac00640 1 -- 172.21.15.163:0/3511551991 shutdown_connections 2024-07-26T06:14:07.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.436+0000 7fbe3ac00640 1 -- 172.21.15.163:0/3511551991 wait complete. 2024-07-26T06:14:07.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.437+0000 7fbe3ac00640 1 Processor -- start 2024-07-26T06:14:07.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.437+0000 7fbe3ac00640 1 -- start start 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.438+0000 7fbe3ac00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 0x7fbe34147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.438+0000 7fbe3ac00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe340cd000 0x7fbe34146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.438+0000 7fbe3ac00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe341469f0 0x7fbe34147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.438+0000 7fbe3ac00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe340cda70 con 0x7fbe340b4420 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.438+0000 7fbe3ac00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3406f580 con 0x7fbe341469f0 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.438+0000 7fbe3ac00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbe340cdbf0 con 0x7fbe340cd000 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 0x7fbe34147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 0x7fbe34147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33634/0 (socket says 172.21.15.163:33634) 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 -- 172.21.15.163:0/1451636254 learned_addr learned my addr 172.21.15.163:0/1451636254 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:07.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 -- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe340cd000 msgr2=0x7fbe34146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe340cd000 0x7fbe34146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 -- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe341469f0 msgr2=0x7fbe34147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe341469f0 0x7fbe34147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 -- 172.21.15.163:0/1451636254 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe34152d40 con 0x7fbe340b4420 2024-07-26T06:14:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.439+0000 7fbe39800640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 0x7fbe34147390 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009950 tx=0x7fbe28032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.440+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28040070 con 0x7fbe340b4420 2024-07-26T06:14:07.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.440+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe28003af0 con 0x7fbe340b4420 2024-07-26T06:14:07.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.440+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe34153020 con 0x7fbe340b4420 2024-07-26T06:14:07.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.440+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28045460 con 0x7fbe340b4420 2024-07-26T06:14:07.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.440+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe340727a0 con 0x7fbe340b4420 2024-07-26T06:14:07.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.441+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe340722c0 con 0x7fbe340b4420 2024-07-26T06:14:07.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.442+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbe28039040 con 0x7fbe340b4420 2024-07-26T06:14:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.443+0000 7fbe31600640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbe14076300 0x7fbe140787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.443+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe280b8de0 con 0x7fbe340b4420 2024-07-26T06:14:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.443+0000 7fbe38e00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbe14076300 0x7fbe140787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:07.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.445+0000 7fbe38e00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbe14076300 0x7fbe140787c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbe341478d0 tx=0x7fbe20007a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:07.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.447+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe28082740 con 0x7fbe340b4420 2024-07-26T06:14:07.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.655+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fbe340cd530 con 0x7fbe340b4420 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.656+0000 7fbe31600640 1 -- 172.21.15.163:0/1451636254 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7fbe28082040 con 0x7fbe340b4420 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:mon advanced mon_op_complaint_time 120 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:mon dev mon_warn_on_pool_no_app false 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd advanced osd_op_complaint_time 180.000000 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 5439.226762 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3901.149445 2024-07-26T06:14:07.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5414.054049 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4219.540367 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4287.026771 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4403.390021 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3932.844298 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4626.659828 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds 20/20 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds_balancer 20/20 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_ms 1/1 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_frag true 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_scatterstat true 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced mds_op_complaint_time 180.000000 2024-07-26T06:14:07.658 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_verify_scatter true 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced osd_op_complaint_time 180.000000 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_mon_op_timeout 900 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_osd_op_timeout 900 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced client_mount_timeout 600 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_client 20/20 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_ms 1/1 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_mon_op_timeout 900 2024-07-26T06:14:07.659 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_osd_op_timeout 900 2024-07-26T06:14:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbe14076300 msgr2=0x7fbe140787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbe14076300 0x7fbe140787c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbe341478d0 tx=0x7fbe20007a30 comp rx=0 tx=0).stop 2024-07-26T06:14:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 msgr2=0x7fbe34147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 0x7fbe34147390 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009950 tx=0x7fbe28032d50 comp rx=0 tx=0).stop 2024-07-26T06:14:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 shutdown_connections 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbe14076300 0x7fbe140787c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe341469f0 0x7fbe34147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe340cd000 0x7fbe34146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 --2- 172.21.15.163:0/1451636254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe340b4420 0x7fbe34147390 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 >> 172.21.15.163:0/1451636254 conn(0x7fbe340b0e30 msgr2=0x7fbe340b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 shutdown_connections 2024-07-26T06:14:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.659+0000 7fbe3ac00640 1 -- 172.21.15.163:0/1451636254 wait complete. 2024-07-26T06:14:07.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.879+0000 7f023d600640 1 -- 172.21.15.163:0/797844726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023815d650 msgr2=0x7f02380d1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.879+0000 7f023d600640 1 --2- 172.21.15.163:0/797844726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023815d650 0x7f02380d1990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f02380745f0 tx=0x7f022802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.879+0000 7f023d600640 1 -- 172.21.15.163:0/797844726 shutdown_connections 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.879+0000 7f023d600640 1 --2- 172.21.15.163:0/797844726 >> v2:172.21.15.164:3300/0 conn(0x7f02380b53b0 0x7f02380b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.879+0000 7f023d600640 1 --2- 172.21.15.163:0/797844726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023815d650 0x7f02380d1990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.879+0000 7f023d600640 1 -- 172.21.15.163:0/797844726 >> 172.21.15.163:0/797844726 conn(0x7f02380d36b0 msgr2=0x7f02380d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.880+0000 7f023d600640 1 -- 172.21.15.163:0/797844726 shutdown_connections 2024-07-26T06:14:07.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.880+0000 7f023d600640 1 -- 172.21.15.163:0/797844726 wait complete. 2024-07-26T06:14:07.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.881+0000 7f023d600640 1 Processor -- start 2024-07-26T06:14:07.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.881+0000 7f023d600640 1 -- start start 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.882+0000 7f023d600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 0x7f02381f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.882+0000 7f023d600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02380b53b0 0x7f02381f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.882+0000 7f023d600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f023815d650 0x7f02381569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.882+0000 7f023d600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f02380d2050 con 0x7f023806c270 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.882+0000 7f023d600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f02380d1ed0 con 0x7f023815d650 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.882+0000 7f023d600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f02380b7730 con 0x7f02380b53b0 2024-07-26T06:14:07.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 0x7f02381f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 0x7f02381f3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33656/0 (socket says 172.21.15.163:33656) 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 -- 172.21.15.163:0/1540188314 learned_addr learned my addr 172.21.15.163:0/1540188314 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f022fe00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02380b53b0 0x7f02381f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0237400640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f023815d650 0x7f02381569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 -- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02380b53b0 msgr2=0x7f02381f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02380b53b0 0x7f02381f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 -- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f023815d650 msgr2=0x7f02381569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f023815d650 0x7f02381569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0236a00640 1 -- 172.21.15.163:0/1540188314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0238154700 con 0x7f023806c270 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f022fe00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02380b53b0 0x7f02381f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.883+0000 7f0237400640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f023815d650 0x7f02381569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:07.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.884+0000 7f0236a00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 0x7f02381f3660 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0228009920 tx=0x7f02280316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:07.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.884+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228033e00 con 0x7f023806c270 2024-07-26T06:14:07.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.884+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0228031a40 con 0x7f023806c270 2024-07-26T06:14:07.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.884+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228031e30 con 0x7f023806c270 2024-07-26T06:14:07.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.884+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02381f4ba0 con 0x7f023806c270 2024-07-26T06:14:07.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.884+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0238154c80 con 0x7f023806c270 2024-07-26T06:14:07.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.885+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02380b5800 con 0x7f023806c270 2024-07-26T06:14:07.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.887+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f02280330d0 con 0x7f023806c270 2024-07-26T06:14:07.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.888+0000 7f022ea00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f02140762d0 0x7f0214078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:07.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.888+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f02280b81f0 con 0x7f023806c270 2024-07-26T06:14:07.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.889+0000 7f022fe00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f02140762d0 0x7f0214078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:07.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.890+0000 7f022fe00640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f02140762d0 0x7f0214078790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f02380c9ef0 tx=0x7f0220005e50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:07.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:07.893+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f022803f030 con 0x7f023806c270 2024-07-26T06:14:08.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.101+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f02380b32a0 con 0x7f023806c270 2024-07-26T06:14:08.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.106+0000 7f022ea00640 1 -- 172.21.15.163:0/1540188314 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f0228081b50 con 0x7f023806c270 2024-07-26T06:14:08.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f02140762d0 msgr2=0x7f0214078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f02140762d0 0x7f0214078790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f02380c9ef0 tx=0x7f0220005e50 comp rx=0 tx=0).stop 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 msgr2=0x7f02381f3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 0x7f02381f3660 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0228009920 tx=0x7f02280316b0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 shutdown_connections 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f02140762d0 0x7f0214078790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f023815d650 0x7f02381569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02380b53b0 0x7f02381f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 --2- 172.21.15.163:0/1540188314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f023806c270 0x7f02381f3660 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.109+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 >> 172.21.15.163:0/1540188314 conn(0x7f02380d36b0 msgr2=0x7f02380ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.110+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 shutdown_connections 2024-07-26T06:14:08.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.110+0000 7f023d600640 1 -- 172.21.15.163:0/1540188314 wait complete. 2024-07-26T06:14:08.129 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:14:08.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:14:08.245 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:14:08.372 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:14:08.373 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:08.373 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:08.432 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:08.432 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:08.487 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:08.487 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:08.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:14:08.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 -- 172.21.15.163:0/1962219251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef94071a70 msgr2=0x7fef940b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 --2- 172.21.15.163:0/1962219251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef94071a70 0x7fef940b3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005fd0 tx=0x7fef8c02a8b0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 -- 172.21.15.163:0/1962219251 shutdown_connections 2024-07-26T06:14:08.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 --2- 172.21.15.163:0/1962219251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef94071a70 0x7fef940b3710 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 --2- 172.21.15.163:0/1962219251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef940ce4a0 0x7fef94071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 -- 172.21.15.163:0/1962219251 >> 172.21.15.163:0/1962219251 conn(0x7fef940b0e30 msgr2=0x7fef940b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:08.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 -- 172.21.15.163:0/1962219251 shutdown_connections 2024-07-26T06:14:08.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.739+0000 7fef93e00640 1 -- 172.21.15.163:0/1962219251 wait complete. 2024-07-26T06:14:08.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.740+0000 7fef93e00640 1 Processor -- start 2024-07-26T06:14:08.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.741+0000 7fef93e00640 1 -- start start 2024-07-26T06:14:08.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.741+0000 7fef93e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef94071a70 0x7fef9414fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef93e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 0x7fef94150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef93e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef941556f0 0x7fef94158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef93e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fef940b4020 con 0x7fef940ce4a0 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef93e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fef940b3ea0 con 0x7fef94071a70 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef93e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fef940b41a0 con 0x7fef941556f0 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 0x7fef94150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 0x7fef94150320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33666/0 (socket says 172.21.15.163:33666) 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 -- 172.21.15.163:0/219764485 learned_addr learned my addr 172.21.15.163:0/219764485 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 -- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef941556f0 msgr2=0x7fef94158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef941556f0 0x7fef94158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 -- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef94071a70 msgr2=0x7fef9414fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef94071a70 0x7fef9414fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.742+0000 7fef8be00640 1 -- 172.21.15.163:0/219764485 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef94156480 con 0x7fef940ce4a0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.743+0000 7fef8be00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 0x7fef94150320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00d770 tx=0x7fef7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.743+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c017070 con 0x7fef940ce4a0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.743+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef7c009d20 con 0x7fef940ce4a0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.743+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0106c0 con 0x7fef940ce4a0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.743+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef941e7880 con 0x7fef940ce4a0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.743+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef941e8120 con 0x7fef940ce4a0 2024-07-26T06:14:08.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.744+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef940b4730 con 0x7fef940ce4a0 2024-07-26T06:14:08.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.745+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fef7c01a070 con 0x7fef940ce4a0 2024-07-26T06:14:08.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.746+0000 7fef8aa00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fef700762d0 0x7fef70078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:08.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.746+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef7c097970 con 0x7fef940ce4a0 2024-07-26T06:14:08.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.747+0000 7fef92a00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fef700762d0 0x7fef70078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:08.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.748+0000 7fef92a00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fef700762d0 0x7fef70078790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef84007fd0 tx=0x7fef84007900 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:08.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.750+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef7c061230 con 0x7fef940ce4a0 2024-07-26T06:14:08.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.977+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fef940ced60 con 0x7fef940ce4a0 2024-07-26T06:14:08.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.978+0000 7fef8aa00640 1 -- 172.21.15.163:0/219764485 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fef7c060b30 con 0x7fef940ce4a0 2024-07-26T06:14:08.979 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.980+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fef700762d0 msgr2=0x7fef70078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.980+0000 7fef93e00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fef700762d0 0x7fef70078790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef84007fd0 tx=0x7fef84007900 comp rx=0 tx=0).stop 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.980+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 msgr2=0x7fef94150320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.980+0000 7fef93e00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 0x7fef94150320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00d770 tx=0x7fef7c00dc30 comp rx=0 tx=0).stop 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 shutdown_connections 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fef700762d0 0x7fef70078790 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef941556f0 0x7fef94158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef940ce4a0 0x7fef94150320 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 --2- 172.21.15.163:0/219764485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef94071a70 0x7fef9414fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:08.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 >> 172.21.15.163:0/219764485 conn(0x7fef940b0e30 msgr2=0x7fef94151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:08.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 shutdown_connections 2024-07-26T06:14:08.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:08.981+0000 7fef93e00640 1 -- 172.21.15.163:0/219764485 wait complete. 2024-07-26T06:14:09.002 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:14:09.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:09.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-07-26T06:14:09.092 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:14:09.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:09.092 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:14:09.150 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:09.150 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:09.208 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:09.208 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:09.261 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:09.261 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:09.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.526+0000 7efd3ec00640 1 -- 172.21.15.163:0/1093544921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 msgr2=0x7efd38073580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.526+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1093544921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd38073580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efd380706e0 tx=0x7efd2002a560 comp rx=0 tx=0).stop 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 -- 172.21.15.163:0/1093544921 shutdown_connections 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1093544921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd38073ac0 0x7efd38070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1093544921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd38073580 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1093544921 >> v2:172.21.15.164:3300/0 conn(0x7efd38140550 0x7efd38141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 -- 172.21.15.163:0/1093544921 >> 172.21.15.163:0/1093544921 conn(0x7efd380b30d0 msgr2=0x7efd380b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 -- 172.21.15.163:0/1093544921 shutdown_connections 2024-07-26T06:14:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.527+0000 7efd3ec00640 1 -- 172.21.15.163:0/1093544921 wait complete. 2024-07-26T06:14:09.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.528+0000 7efd3ec00640 1 Processor -- start 2024-07-26T06:14:09.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 -- start start 2024-07-26T06:14:09.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd38073ac0 0x7efd38158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd38140550 0x7efd381541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd381546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd38071370 con 0x7efd38140930 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efd380711f0 con 0x7efd38073ac0 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.529+0000 7efd3ec00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd380714f0 con 0x7efd38140550 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd381546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd381546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33684/0 (socket says 172.21.15.163:33684) 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 -- 172.21.15.163:0/1701737431 learned_addr learned my addr 172.21.15.163:0/1701737431 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd2fe00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd38140550 0x7efd381541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:09.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd37e00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd38073ac0 0x7efd38158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 -- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd38140550 msgr2=0x7efd381541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd38140550 0x7efd381541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 -- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd38073ac0 msgr2=0x7efd38158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd38073ac0 0x7efd38158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.530+0000 7efd3cc00640 1 -- 172.21.15.163:0/1701737431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd38155f60 con 0x7efd38140930 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.531+0000 7efd3cc00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd381546f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efd3000eaa0 tx=0x7efd3000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.531+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3000cdb0 con 0x7efd38140930 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.531+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd30004c10 con 0x7efd38140930 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.531+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30005020 con 0x7efd38140930 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.531+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd38156240 con 0x7efd38140930 2024-07-26T06:14:09.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.531+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd38156800 con 0x7efd38140930 2024-07-26T06:14:09.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.532+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd38071a80 con 0x7efd38140930 2024-07-26T06:14:09.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.533+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efd300051c0 con 0x7efd38140930 2024-07-26T06:14:09.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.534+0000 7efd34c00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efd140762d0 0x7efd14078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:09.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.534+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efd30098120 con 0x7efd38140930 2024-07-26T06:14:09.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.535+0000 7efd37e00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efd140762d0 0x7efd14078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:09.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.536+0000 7efd37e00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efd140762d0 0x7efd14078790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd28005fd0 tx=0x7efd28005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:09.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.540+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd30016030 con 0x7efd38140930 2024-07-26T06:14:09.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.767+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efd3813f990 con 0x7efd38140930 2024-07-26T06:14:09.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.768+0000 7efd34c00640 1 -- 172.21.15.163:0/1701737431 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7efd30061a80 con 0x7efd38140930 2024-07-26T06:14:09.769 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efd140762d0 msgr2=0x7efd14078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efd140762d0 0x7efd14078790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd28005fd0 tx=0x7efd28005850 comp rx=0 tx=0).stop 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 msgr2=0x7efd381546f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd381546f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efd3000eaa0 tx=0x7efd3000ef60 comp rx=0 tx=0).stop 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 shutdown_connections 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efd140762d0 0x7efd14078790 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd38140930 0x7efd381546f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd38140550 0x7efd381541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 --2- 172.21.15.163:0/1701737431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd38073ac0 0x7efd38158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.771+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 >> 172.21.15.163:0/1701737431 conn(0x7efd380b30d0 msgr2=0x7efd3815af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.772+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 shutdown_connections 2024-07-26T06:14:09.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:09.772+0000 7efd3ec00640 1 -- 172.21.15.163:0/1701737431 wait complete. 2024-07-26T06:14:09.791 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:14:09.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:09.791 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-07-26T06:14:09.883 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:14:09.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:09.883 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:14:09.937 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:09.938 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:09.994 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:09.994 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:10.048 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:10.049 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:10.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.312+0000 7f1c39400640 1 -- 172.21.15.163:0/3387381254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c3415a1d0 msgr2=0x7f1c3406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.312+0000 7f1c39400640 1 --2- 172.21.15.163:0/3387381254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c3415a1d0 0x7f1c3406fef0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009830 tx=0x7f1c1c02a510 comp rx=0 tx=0).stop 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.313+0000 7f1c39400640 1 -- 172.21.15.163:0/3387381254 shutdown_connections 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.313+0000 7f1c39400640 1 --2- 172.21.15.163:0/3387381254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c340aefc0 0x7f1c34071a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.313+0000 7f1c39400640 1 --2- 172.21.15.163:0/3387381254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c3415a1d0 0x7f1c3406fef0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.313+0000 7f1c39400640 1 -- 172.21.15.163:0/3387381254 >> 172.21.15.163:0/3387381254 conn(0x7f1c340cc400 msgr2=0x7f1c340cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.313+0000 7f1c39400640 1 -- 172.21.15.163:0/3387381254 shutdown_connections 2024-07-26T06:14:10.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.313+0000 7f1c39400640 1 -- 172.21.15.163:0/3387381254 wait complete. 2024-07-26T06:14:10.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.315+0000 7f1c39400640 1 Processor -- start 2024-07-26T06:14:10.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.315+0000 7f1c39400640 1 -- start start 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.316+0000 7f1c39400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34071f90 0x7f1c3414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.316+0000 7f1c39400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 0x7f1c3415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.316+0000 7f1c39400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c3415a1d0 0x7f1c3415d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.316+0000 7f1c39400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3406d0d0 con 0x7f1c340aefc0 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.316+0000 7f1c39400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3406cf50 con 0x7f1c3415a1d0 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.316+0000 7f1c39400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3406d250 con 0x7f1c34071f90 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 0x7f1c3415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 0x7f1c3415cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33692/0 (socket says 172.21.15.163:33692) 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 -- 172.21.15.163:0/2727373958 learned_addr learned my addr 172.21.15.163:0/2727373958 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:10.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33e00640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34071f90 0x7f1c3414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 -- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34071f90 msgr2=0x7f1c3414eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34071f90 0x7f1c3414eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 -- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c3415a1d0 msgr2=0x7f1c3415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c3415a1d0 0x7f1c3415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 -- 172.21.15.163:0/2727373958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c340c5930 con 0x7f1c340aefc0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.317+0000 7f1c33400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 0x7f1c3415cf50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400ea80 tx=0x7f1c2400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.318+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400ce40 con 0x7f1c340aefc0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.318+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c24011560 con 0x7f1c340aefc0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.318+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24011950 con 0x7f1c340aefc0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.318+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c340c5c10 con 0x7f1c340aefc0 2024-07-26T06:14:10.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.318+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3415abb0 con 0x7f1c340aefc0 2024-07-26T06:14:10.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.319+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3406d7e0 con 0x7f1c340aefc0 2024-07-26T06:14:10.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.320+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1c24011af0 con 0x7f1c340aefc0 2024-07-26T06:14:10.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.321+0000 7f1c2be00640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c100762d0 0x7f1c10078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:10.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.321+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1c24098430 con 0x7f1c340aefc0 2024-07-26T06:14:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.322+0000 7f1c33e00640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c100762d0 0x7f1c10078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:10.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.323+0000 7f1c33e00640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c100762d0 0x7f1c10078790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c034000 tx=0x7f1c1c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.325+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c24016030 con 0x7f1c340aefc0 2024-07-26T06:14:10.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.548+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1c340aea10 con 0x7f1c340aefc0 2024-07-26T06:14:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.548+0000 7f1c2be00640 1 -- 172.21.15.163:0/2727373958 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f1c24061cf0 con 0x7f1c340aefc0 2024-07-26T06:14:10.549 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:14:10.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c100762d0 msgr2=0x7f1c10078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:10.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c100762d0 0x7f1c10078790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c034000 tx=0x7f1c1c004730 comp rx=0 tx=0).stop 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 msgr2=0x7f1c3415cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 0x7f1c3415cf50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400ea80 tx=0x7f1c2400ef40 comp rx=0 tx=0).stop 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 shutdown_connections 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1c100762d0 0x7f1c10078790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c3415a1d0 0x7f1c3415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1c340aefc0 0x7f1c3415cf50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 --2- 172.21.15.163:0/2727373958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c34071f90 0x7f1c3414eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.551+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 >> 172.21.15.163:0/2727373958 conn(0x7f1c340cc400 msgr2=0x7f1c340b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.552+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 shutdown_connections 2024-07-26T06:14:10.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:10.552+0000 7f1c39400640 1 -- 172.21.15.163:0/2727373958 wait complete. 2024-07-26T06:14:10.571 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-07-26T06:14:10.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:10.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-07-26T06:14:10.646 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:14:10.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:10.647 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-07-26T06:14:10.694 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:10.694 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:10.752 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:10.752 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:10.806 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:10.806 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:10.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:14:11.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 -- 172.21.15.163:0/2222502101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d00ce830 msgr2=0x7f26d00b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 --2- 172.21.15.163:0/2222502101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d00ce830 0x7f26d00b64c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009830 tx=0x7f26c002a400 comp rx=0 tx=0).stop 2024-07-26T06:14:11.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 -- 172.21.15.163:0/2222502101 shutdown_connections 2024-07-26T06:14:11.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 --2- 172.21.15.163:0/2222502101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d00ce830 0x7f26d00b64c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 --2- 172.21.15.163:0/2222502101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26d015d5a0 0x7f26d00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 -- 172.21.15.163:0/2222502101 >> 172.21.15.163:0/2222502101 conn(0x7f26d00cfda0 msgr2=0x7f26d00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:11.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 -- 172.21.15.163:0/2222502101 shutdown_connections 2024-07-26T06:14:11.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.074+0000 7f26d7200640 1 -- 172.21.15.163:0/2222502101 wait complete. 2024-07-26T06:14:11.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.076+0000 7f26d7200640 1 Processor -- start 2024-07-26T06:14:11.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.076+0000 7f26d7200640 1 -- start start 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d7200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26d00b1e80 0x7f26d01f7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d7200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26d00ce830 0x7f26d00b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d7200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 0x7f26d00b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d7200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26d00b5130 con 0x7f26d015d5a0 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d7200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f26d00b4fb0 con 0x7f26d00ce830 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d7200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f26d00b52b0 con 0x7f26d00b1e80 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 0x7f26d00b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:11.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 0x7f26d00b7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33710/0 (socket says 172.21.15.163:33710) 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 -- 172.21.15.163:0/1704210248 learned_addr learned my addr 172.21.15.163:0/1704210248 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 -- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26d00b1e80 msgr2=0x7f26d01f7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26d00b1e80 0x7f26d01f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 -- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26d00ce830 msgr2=0x7f26d00b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26d00ce830 0x7f26d00b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.077+0000 7f26d6800640 1 -- 172.21.15.163:0/1704210248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d0152ea0 con 0x7f26d015d5a0 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.078+0000 7f26d6800640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 0x7f26d00b7fc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00c770 tx=0x7f26cc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.078+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc003da0 con 0x7f26d015d5a0 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.078+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26cc005070 con 0x7f26d015d5a0 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.078+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc005500 con 0x7f26d015d5a0 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.078+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d00c9360 con 0x7f26d015d5a0 2024-07-26T06:14:11.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.078+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d00c9920 con 0x7f26d015d5a0 2024-07-26T06:14:11.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.079+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d00cf860 con 0x7f26d015d5a0 2024-07-26T06:14:11.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.081+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26cc0056a0 con 0x7f26d015d5a0 2024-07-26T06:14:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.082+0000 7f26ca000640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26b40762d0 0x7f26b4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.082+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f26cc097710 con 0x7f26d015d5a0 2024-07-26T06:14:11.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.082+0000 7f26d5e00640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26b40762d0 0x7f26b4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.083+0000 7f26d5e00640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26b40762d0 0x7f26b4078790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f26c400a8f0 tx=0x7f26c4005bd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.086+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26d00cf860 con 0x7f26d015d5a0 2024-07-26T06:14:11.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.305+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f26d00b40a0 con 0x7f26d015d5a0 2024-07-26T06:14:11.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.306+0000 7f26ca000640 1 -- 172.21.15.163:0/1704210248 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f26d00b40a0 con 0x7f26d015d5a0 2024-07-26T06:14:11.307 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:14:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26b40762d0 msgr2=0x7f26b4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26b40762d0 0x7f26b4078790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f26c400a8f0 tx=0x7f26c4005bd0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 msgr2=0x7f26d00b7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 0x7f26d00b7fc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00c770 tx=0x7f26cc00cc30 comp rx=0 tx=0).stop 2024-07-26T06:14:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 shutdown_connections 2024-07-26T06:14:11.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26b40762d0 0x7f26b4078790 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26d015d5a0 0x7f26d00b7fc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26d00ce830 0x7f26d00b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 --2- 172.21.15.163:0/1704210248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26d00b1e80 0x7f26d01f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 >> 172.21.15.163:0/1704210248 conn(0x7f26d00cfda0 msgr2=0x7f26d00b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 shutdown_connections 2024-07-26T06:14:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.309+0000 7f26d7200640 1 -- 172.21.15.163:0/1704210248 wait complete. 2024-07-26T06:14:11.330 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-07-26T06:14:11.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:11.330 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-07-26T06:14:11.426 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:14:11.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:11.427 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-07-26T06:14:11.480 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-07-26T06:14:11.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:11.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 -- 172.21.15.163:0/1373820940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 msgr2=0x7f5140004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 --2- 172.21.15.163:0/1373820940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f5140004c80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5144001690 tx=0x7f5140005830 comp rx=0 tx=0).stop 2024-07-26T06:14:11.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 -- 172.21.15.163:0/1373820940 shutdown_connections 2024-07-26T06:14:11.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 --2- 172.21.15.163:0/1373820940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f5140004c80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 -- 172.21.15.163:0/1373820940 >> 172.21.15.163:0/1373820940 conn(0x7f515c0cfda0 msgr2=0x7f515c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:11.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 -- 172.21.15.163:0/1373820940 shutdown_connections 2024-07-26T06:14:11.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.691+0000 7f515b200640 1 -- 172.21.15.163:0/1373820940 wait complete. 2024-07-26T06:14:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.693+0000 7f515b200640 1 Processor -- start 2024-07-26T06:14:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.693+0000 7f515b200640 1 -- start start 2024-07-26T06:14:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f515b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f515c1f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f515b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f515c149570 0x7f515c0b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f515b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f515c0cc6c0 0x7f515c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f515b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f515c0ce470 con 0x7f5140004800 2024-07-26T06:14:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f515b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f515c0ce2f0 con 0x7f515c149570 2024-07-26T06:14:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f515b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f515c0ce5f0 con 0x7f515c0cc6c0 2024-07-26T06:14:11.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f5159e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f515c1f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:11.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f5159e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f515c1f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33726/0 (socket says 172.21.15.163:33726) 2024-07-26T06:14:11.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.694+0000 7f5159e00640 1 -- 172.21.15.163:0/443531647 learned_addr learned my addr 172.21.15.163:0/443531647 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:11.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5159e00640 1 -- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f515c0cc6c0 msgr2=0x7f515c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f515a800640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f515c0cc6c0 0x7f515c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:11.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5153e00640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f515c149570 0x7f515c0b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:11.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5159e00640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f515c0cc6c0 0x7f515c0ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5159e00640 1 -- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f515c149570 msgr2=0x7f515c0b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5159e00640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f515c149570 0x7f515c0b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5159e00640 1 -- 172.21.15.163:0/443531647 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c1554b0 con 0x7f5140004800 2024-07-26T06:14:11.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5159e00640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f515c1f3a60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f514c00e740 tx=0x7f514c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c00ccc0 con 0x7f5140004800 2024-07-26T06:14:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f514c00ce60 con 0x7f5140004800 2024-07-26T06:14:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c005280 con 0x7f5140004800 2024-07-26T06:14:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.695+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f515c0ca590 con 0x7f5140004800 2024-07-26T06:14:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.696+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f515c0cab50 con 0x7f5140004800 2024-07-26T06:14:11.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.697+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515c0b6740 con 0x7f5140004800 2024-07-26T06:14:11.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.698+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f514c005420 con 0x7f5140004800 2024-07-26T06:14:11.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.699+0000 7f5152000640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51380762d0 0x7f5138078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:11.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.699+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f514c098060 con 0x7f5140004800 2024-07-26T06:14:11.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.699+0000 7f5153e00640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51380762d0 0x7f5138078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:11.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.701+0000 7f5153e00640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51380762d0 0x7f5138078790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5144032000 tx=0x7f5144001190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:11.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.703+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f514c061920 con 0x7f5140004800 2024-07-26T06:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.949+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f515c15be00 con 0x7f5140004800 2024-07-26T06:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.950+0000 7f5152000640 1 -- 172.21.15.163:0/443531647 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7f514c061220 con 0x7f5140004800 2024-07-26T06:14:11.952 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:11.952 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:14:11.952 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51380762d0 msgr2=0x7f5138078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51380762d0 0x7f5138078790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5144032000 tx=0x7f5144001190 comp rx=0 tx=0).stop 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 msgr2=0x7f515c1f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f515c1f3a60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f514c00e740 tx=0x7f514c00ec00 comp rx=0 tx=0).stop 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 shutdown_connections 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51380762d0 0x7f5138078790 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f515c0cc6c0 0x7f515c0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f515c149570 0x7f515c0b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 --2- 172.21.15.163:0/443531647 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140004800 0x7f515c1f3a60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 >> 172.21.15.163:0/443531647 conn(0x7f515c0cfda0 msgr2=0x7f515c159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 shutdown_connections 2024-07-26T06:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:11.953+0000 7f515b200640 1 -- 172.21.15.163:0/443531647 wait complete. 2024-07-26T06:14:11.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:12.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 -- 172.21.15.163:0/3027826191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c0b2f90 msgr2=0x7fdc1c159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 --2- 172.21.15.163:0/3027826191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c0b2f90 0x7fdc1c159380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdc08005fd0 tx=0x7fdc0802aa00 comp rx=0 tx=0).stop 2024-07-26T06:14:12.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 -- 172.21.15.163:0/3027826191 shutdown_connections 2024-07-26T06:14:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 --2- 172.21.15.163:0/3027826191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc1c06bf40 0x7fdc1c06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 --2- 172.21.15.163:0/3027826191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c0b2f90 0x7fdc1c159380 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 -- 172.21.15.163:0/3027826191 >> 172.21.15.163:0/3027826191 conn(0x7fdc1c06d420 msgr2=0x7fdc1c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 -- 172.21.15.163:0/3027826191 shutdown_connections 2024-07-26T06:14:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.180+0000 7fdc21a00640 1 -- 172.21.15.163:0/3027826191 wait complete. 2024-07-26T06:14:12.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.182+0000 7fdc21a00640 1 Processor -- start 2024-07-26T06:14:12.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.182+0000 7fdc21a00640 1 -- start start 2024-07-26T06:14:12.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc21a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 0x7fdc1c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc21a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc1c06f350 0x7fdc1c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc21a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc1c0b2f90 0x7fdc1c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc21a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c164280 con 0x7fdc1c06bf40 2024-07-26T06:14:12.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc21a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c0cdc10 con 0x7fdc1c0b2f90 2024-07-26T06:14:12.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc21a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c164400 con 0x7fdc1c06f350 2024-07-26T06:14:12.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc1aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 0x7fdc1c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc1aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 0x7fdc1c0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33746/0 (socket says 172.21.15.163:33746) 2024-07-26T06:14:12.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.183+0000 7fdc1aa00640 1 -- 172.21.15.163:0/573454123 learned_addr learned my addr 172.21.15.163:0/573454123 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:12.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1a000640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc1c06f350 0x7fdc1c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1aa00640 1 -- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc1c06f350 msgr2=0x7fdc1c1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1b400640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc1c0b2f90 0x7fdc1c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1aa00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc1c06f350 0x7fdc1c1582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1aa00640 1 -- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc1c0b2f90 msgr2=0x7fdc1c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1aa00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc1c0b2f90 0x7fdc1c0ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1aa00640 1 -- 172.21.15.163:0/573454123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c156db0 con 0x7fdc1c06bf40 2024-07-26T06:14:12.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc1aa00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 0x7fdc1c0b19f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00c450 tx=0x7fdc0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:12.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.184+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c017070 con 0x7fdc1c06bf40 2024-07-26T06:14:12.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.185+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc0c0053c0 con 0x7fdc1c06bf40 2024-07-26T06:14:12.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.185+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c003850 con 0x7fdc1c06bf40 2024-07-26T06:14:12.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.185+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c157090 con 0x7fdc1c06bf40 2024-07-26T06:14:12.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.185+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c157650 con 0x7fdc1c06bf40 2024-07-26T06:14:12.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.186+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1c06c380 con 0x7fdc1c06bf40 2024-07-26T06:14:12.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.187+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc0c01a070 con 0x7fdc1c06bf40 2024-07-26T06:14:12.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.188+0000 7fdc12a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdbfc0762d0 0x7fdbfc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.188+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdc0c096c00 con 0x7fdc1c06bf40 2024-07-26T06:14:12.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.189+0000 7fdc1a000640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdbfc0762d0 0x7fdbfc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.190+0000 7fdc1a000640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdbfc0762d0 0x7fdbfc078790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc08035000 tx=0x7fdc08004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:12.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.192+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc0c060550 con 0x7fdc1c06bf40 2024-07-26T06:14:12.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.443+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc1c153dc0 con 0x7fdc1c06bf40 2024-07-26T06:14:12.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.444+0000 7fdc12a00640 1 -- 172.21.15.163:0/573454123 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7fdc0c05fe50 con 0x7fdc1c06bf40 2024-07-26T06:14:12.445 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:12.445 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:14:12.445 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:14:12.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.446+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdbfc0762d0 msgr2=0x7fdbfc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.446+0000 7fdc21a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdbfc0762d0 0x7fdbfc078790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc08035000 tx=0x7fdc08004ab0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.446+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 msgr2=0x7fdc1c0b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.446+0000 7fdc21a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 0x7fdc1c0b19f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00c450 tx=0x7fdc0c00c910 comp rx=0 tx=0).stop 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 shutdown_connections 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdbfc0762d0 0x7fdbfc078790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc1c0b2f90 0x7fdc1c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc1c06f350 0x7fdc1c1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 --2- 172.21.15.163:0/573454123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc1c06bf40 0x7fdc1c0b19f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 >> 172.21.15.163:0/573454123 conn(0x7fdc1c06d420 msgr2=0x7fdc1c072e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:12.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 shutdown_connections 2024-07-26T06:14:12.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.447+0000 7fdc21a00640 1 -- 172.21.15.163:0/573454123 wait complete. 2024-07-26T06:14:12.469 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:12.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 -- 172.21.15.163:0/58071656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff38004800 msgr2=0x7eff38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 --2- 172.21.15.163:0/58071656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff38004800 0x7eff38004c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff3c006fd0 tx=0x7eff38005830 comp rx=0 tx=0).stop 2024-07-26T06:14:12.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 -- 172.21.15.163:0/58071656 shutdown_connections 2024-07-26T06:14:12.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 --2- 172.21.15.163:0/58071656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff38004800 0x7eff38004c80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 -- 172.21.15.163:0/58071656 >> 172.21.15.163:0/58071656 conn(0x7eff540cfda0 msgr2=0x7eff540d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:12.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 -- 172.21.15.163:0/58071656 shutdown_connections 2024-07-26T06:14:12.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.677+0000 7eff59a00640 1 -- 172.21.15.163:0/58071656 wait complete. 2024-07-26T06:14:12.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.679+0000 7eff59a00640 1 Processor -- start 2024-07-26T06:14:12.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.679+0000 7eff59a00640 1 -- start start 2024-07-26T06:14:12.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff38004800 0x7eff54153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff541f3910 0x7eff540cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 0x7eff540c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff540ce470 con 0x7eff540c8260 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7eff540ce2f0 con 0x7eff541f3910 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7eff540ce5f0 con 0x7eff38004800 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 0x7eff540c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 0x7eff540c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33768/0 (socket says 172.21.15.163:33768) 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 -- 172.21.15.163:0/743154147 learned_addr learned my addr 172.21.15.163:0/743154147 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff53e00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff38004800 0x7eff54153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff53400640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff541f3910 0x7eff540cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 -- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff38004800 msgr2=0x7eff54153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff38004800 0x7eff54153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 -- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff541f3910 msgr2=0x7eff540cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff541f3910 0x7eff540cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.680+0000 7eff59000640 1 -- 172.21.15.163:0/743154147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff540cba20 con 0x7eff540c8260 2024-07-26T06:14:12.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.681+0000 7eff59000640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 0x7eff540c86e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff4c00d410 tx=0x7eff4c00d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:12.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.681+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c017070 con 0x7eff540c8260 2024-07-26T06:14:12.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.681+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff4c005030 con 0x7eff540c8260 2024-07-26T06:14:12.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.681+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c005460 con 0x7eff540c8260 2024-07-26T06:14:12.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.681+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff540cbd00 con 0x7eff540c8260 2024-07-26T06:14:12.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.681+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff540cc2c0 con 0x7eff540c8260 2024-07-26T06:14:12.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.682+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff540b6740 con 0x7eff540c8260 2024-07-26T06:14:12.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.684+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff4c01a070 con 0x7eff540c8260 2024-07-26T06:14:12.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.685+0000 7eff4be00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff30076320 0x7eff300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:12.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.685+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff4c096670 con 0x7eff540c8260 2024-07-26T06:14:12.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.686+0000 7eff53e00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff30076320 0x7eff300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:12.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.687+0000 7eff53e00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff30076320 0x7eff300787e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7eff5406b090 tx=0x7eff44009500 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:12.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.691+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff4c05ffb0 con 0x7eff540c8260 2024-07-26T06:14:12.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.899+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff5415ca50 con 0x7eff540c8260 2024-07-26T06:14:12.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.901+0000 7eff4be00640 1 -- 172.21.15.163:0/743154147 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16629 (secure 0 0 0) 0x7eff4c05f8b0 con 0x7eff540c8260 2024-07-26T06:14:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:12.902 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":21,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:13:36.040341+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff30076320 msgr2=0x7eff300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff30076320 0x7eff300787e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7eff5406b090 tx=0x7eff44009500 comp rx=0 tx=0).stop 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 msgr2=0x7eff540c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 0x7eff540c86e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff4c00d410 tx=0x7eff4c00d810 comp rx=0 tx=0).stop 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 shutdown_connections 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7eff30076320 0x7eff300787e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff540c8260 0x7eff540c86e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff541f3910 0x7eff540cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 --2- 172.21.15.163:0/743154147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff38004800 0x7eff54153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:12.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 >> 172.21.15.163:0/743154147 conn(0x7eff540cfda0 msgr2=0x7eff540d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:12.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 shutdown_connections 2024-07-26T06:14:12.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:12.904+0000 7eff59a00640 1 -- 172.21.15.163:0/743154147 wait complete. 2024-07-26T06:14:12.926 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:14:12.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 -- 172.21.15.163:0/854672636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 msgr2=0x7fa0c00ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 --2- 172.21.15.163:0/854672636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 0x7fa0c00ce1c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8005fd0 tx=0x7fa0b802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 -- 172.21.15.163:0/854672636 shutdown_connections 2024-07-26T06:14:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 --2- 172.21.15.163:0/854672636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 0x7fa0c00ce1c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 --2- 172.21.15.163:0/854672636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c006bf40 0x7fa0c006c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 -- 172.21.15.163:0/854672636 >> 172.21.15.163:0/854672636 conn(0x7fa0c006d420 msgr2=0x7fa0c006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 -- 172.21.15.163:0/854672636 shutdown_connections 2024-07-26T06:14:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.131+0000 7fa0c6400640 1 -- 172.21.15.163:0/854672636 wait complete. 2024-07-26T06:14:13.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.133+0000 7fa0c6400640 1 Processor -- start 2024-07-26T06:14:13.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.134+0000 7fa0c6400640 1 -- start start 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0c6400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c006bf40 0x7fa0c01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0c6400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 0x7fa0c01e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0c6400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 0x7fa0c01e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0c6400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c00ce700 con 0x7fa0c01404a0 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0c6400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c0161270 con 0x7fa0c00cdd10 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0c6400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c00ce880 con 0x7fa0c006bf40 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0bfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 0x7fa0c01e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0bfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 0x7fa0c01e72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33780/0 (socket says 172.21.15.163:33780) 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0bfe00640 1 -- 172.21.15.163:0/492294284 learned_addr learned my addr 172.21.15.163:0/492294284 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.135+0000 7fa0bf400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c006bf40 0x7fa0c01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0b6a00640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 0x7fa0c01e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0bfe00640 1 -- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c006bf40 msgr2=0x7fa0c01573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0bfe00640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c006bf40 0x7fa0c01573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0bfe00640 1 -- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 msgr2=0x7fa0c01e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0bfe00640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 0x7fa0c01e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0bfe00640 1 -- 172.21.15.163:0/492294284 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c0130270 con 0x7fa0c01404a0 2024-07-26T06:14:13.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0bfe00640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 0x7fa0c01e72a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8006010 tx=0x7fa0b8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b803f070 con 0x7fa0c01404a0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0b8003e10 con 0x7fa0c01404a0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c0157910 con 0x7fa0c01404a0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.136+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8035030 con 0x7fa0c01404a0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.137+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c00b1760 con 0x7fa0c01404a0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.138+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c00ced20 con 0x7fa0c01404a0 2024-07-26T06:14:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.139+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa0b8042070 con 0x7fa0c01404a0 2024-07-26T06:14:13.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.140+0000 7fa0b7e00640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa09c0762d0 0x7fa09c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.141+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa0b80b8320 con 0x7fa0c01404a0 2024-07-26T06:14:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.141+0000 7fa0bf400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa09c0762d0 0x7fa09c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.142+0000 7fa0bf400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa09c0762d0 0x7fa09c078790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001ce0 tx=0x7fa0b0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.145+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0b8081be0 con 0x7fa0c01404a0 2024-07-26T06:14:13.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.386+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0c006cd80 con 0x7fa0c01404a0 2024-07-26T06:14:13.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.388+0000 7fa0b7e00640 1 -- 172.21.15.163:0/492294284 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7fa0b80814e0 con 0x7fa0c01404a0 2024-07-26T06:14:13.389 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:13.389 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:14:13.389 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:14:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa09c0762d0 msgr2=0x7fa09c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa09c0762d0 0x7fa09c078790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001ce0 tx=0x7fa0b0009040 comp rx=0 tx=0).stop 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 msgr2=0x7fa0c01e72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 0x7fa0c01e72a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8006010 tx=0x7fa0b8003820 comp rx=0 tx=0).stop 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 shutdown_connections 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa09c0762d0 0x7fa09c078790 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0c01404a0 0x7fa0c01e72a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c00cdd10 0x7fa0c01e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 --2- 172.21.15.163:0/492294284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c006bf40 0x7fa0c01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.390+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 >> 172.21.15.163:0/492294284 conn(0x7fa0c006d420 msgr2=0x7fa0c0161770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.391+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 shutdown_connections 2024-07-26T06:14:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.391+0000 7fa0c6400640 1 -- 172.21.15.163:0/492294284 wait complete. 2024-07-26T06:14:13.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:13.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 -- 172.21.15.163:0/1742073156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe03415d5a0 msgr2=0x7fe0340ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 --2- 172.21.15.163:0/1742073156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe03415d5a0 0x7fe0340ce2f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe028005fd0 tx=0x7fe02802a540 comp rx=0 tx=0).stop 2024-07-26T06:14:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 -- 172.21.15.163:0/1742073156 shutdown_connections 2024-07-26T06:14:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 --2- 172.21.15.163:0/1742073156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe03415d5a0 0x7fe0340ce2f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 -- 172.21.15.163:0/1742073156 >> 172.21.15.163:0/1742073156 conn(0x7fe0340cfda0 msgr2=0x7fe0340d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 -- 172.21.15.163:0/1742073156 shutdown_connections 2024-07-26T06:14:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.613+0000 7fe03a600640 1 -- 172.21.15.163:0/1742073156 wait complete. 2024-07-26T06:14:13.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.615+0000 7fe03a600640 1 Processor -- start 2024-07-26T06:14:13.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.615+0000 7fe03a600640 1 -- start start 2024-07-26T06:14:13.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe03a600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 0x7fe0340ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe03a600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0340ce830 0x7fe034145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe03a600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe03415d5a0 0x7fe034146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe03a600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe0340b6f60 con 0x7fe0340b1e80 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe03a600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe0340b6de0 con 0x7fe0340ce830 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe03a600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe0340b54b0 con 0x7fe03415d5a0 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe039200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 0x7fe0340ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe039200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 0x7fe0340ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33794/0 (socket says 172.21.15.163:33794) 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.616+0000 7fe039200640 1 -- 172.21.15.163:0/3325747317 learned_addr learned my addr 172.21.15.163:0/3325747317 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe039200640 1 -- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe03415d5a0 msgr2=0x7fe034146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe033e00640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0340ce830 0x7fe034145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe039200640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe03415d5a0 0x7fe034146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe039200640 1 -- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0340ce830 msgr2=0x7fe034145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe039200640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0340ce830 0x7fe034145f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe039200640 1 -- 172.21.15.163:0/3325747317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0340caa80 con 0x7fe0340b1e80 2024-07-26T06:14:13.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe039200640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 0x7fe0340ca540 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe028034040 tx=0x7fe0280039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:13.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02803e030 con 0x7fe0340b1e80 2024-07-26T06:14:13.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0280047a0 con 0x7fe0340b1e80 2024-07-26T06:14:13.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0340c89e0 con 0x7fe0340b1e80 2024-07-26T06:14:13.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028035040 con 0x7fe0340b1e80 2024-07-26T06:14:13.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.617+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0340c8fa0 con 0x7fe0340b1e80 2024-07-26T06:14:13.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.618+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0340ce2f0 con 0x7fe0340b1e80 2024-07-26T06:14:13.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.620+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe028041030 con 0x7fe0340b1e80 2024-07-26T06:14:13.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.620+0000 7fe030c00640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe0140762d0 0x7fe014078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:13.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.621+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe0280b7fe0 con 0x7fe0340b1e80 2024-07-26T06:14:13.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.621+0000 7fe033e00640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe0140762d0 0x7fe014078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:13.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.622+0000 7fe033e00640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe0140762d0 0x7fe014078790 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe020001640 tx=0x7fe020009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:13.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.627+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe028081860 con 0x7fe0340b1e80 2024-07-26T06:14:13.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.854+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe03415c9e0 con 0x7fe0340b1e80 2024-07-26T06:14:13.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.855+0000 7fe030c00640 1 -- 172.21.15.163:0/3325747317 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5229 (secure 0 0 0) 0x7fe028081160 con 0x7fe0340b1e80 2024-07-26T06:14:13.856 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:13.856 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":8,"btime":"2024-07-26T06:13:38:066894+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4167276527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4167276527},{"type":"v1","addr":"172.21.15.164:6836","nonce":4167276527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4335,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3568597782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3568597782},{"type":"v1","addr":"172.21.15.163:6833","nonce":3568597782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4353,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3204700868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3204700868},{"type":"v1","addr":"172.21.15.164:6837","nonce":3204700868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:13:34.510631+0000","modified":"2024-07-26T06:13:37.550193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4372},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4372":{"gid":4372,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/2923632545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":2923632545},{"type":"v1","addr":"172.21.15.163:6835","nonce":2923632545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4372,"qdb_cluster":[4372]},"id":1}]} 2024-07-26T06:14:13.857 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 8 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.857+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe0140762d0 msgr2=0x7fe014078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.857+0000 7fe03a600640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe0140762d0 0x7fe014078790 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe020001640 tx=0x7fe020009040 comp rx=0 tx=0).stop 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 msgr2=0x7fe0340ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 0x7fe0340ca540 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe028034040 tx=0x7fe0280039e0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 shutdown_connections 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe0140762d0 0x7fe014078790 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe03415d5a0 0x7fe034146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0340ce830 0x7fe034145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 --2- 172.21.15.163:0/3325747317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0340b1e80 0x7fe0340ca540 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:13.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 >> 172.21.15.163:0/3325747317 conn(0x7fe0340cfda0 msgr2=0x7fe0340b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:13.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 shutdown_connections 2024-07-26T06:14:13.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:13.858+0000 7fe03a600640 1 -- 172.21.15.163:0/3325747317 wait complete. 2024-07-26T06:14:13.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:14.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 -- 172.21.15.163:0/4242594865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2041556f0 msgr2=0x7fd2040cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 --2- 172.21.15.163:0/4242594865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2041556f0 0x7fd2040cdd90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd2040cea70 tx=0x7fd1f802a560 comp rx=0 tx=0).stop 2024-07-26T06:14:14.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 -- 172.21.15.163:0/4242594865 shutdown_connections 2024-07-26T06:14:14.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 --2- 172.21.15.163:0/4242594865 >> v2:172.21.15.164:3300/0 conn(0x7fd204071a70 0x7fd20415ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 --2- 172.21.15.163:0/4242594865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2040ce2d0 0x7fd204071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 --2- 172.21.15.163:0/4242594865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2041556f0 0x7fd2040cdd90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 -- 172.21.15.163:0/4242594865 >> 172.21.15.163:0/4242594865 conn(0x7fd2040b0e30 msgr2=0x7fd2040b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:14.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 -- 172.21.15.163:0/4242594865 shutdown_connections 2024-07-26T06:14:14.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.067+0000 7fd20aa00640 1 -- 172.21.15.163:0/4242594865 wait complete. 2024-07-26T06:14:14.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.069+0000 7fd20aa00640 1 Processor -- start 2024-07-26T06:14:14.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.069+0000 7fd20aa00640 1 -- start start 2024-07-26T06:14:14.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 0x7fd2040a9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20aa00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2040ce2d0 0x7fd2041e6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20aa00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2041556f0 0x7fd2041e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20aa00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd20415f2c0 con 0x7fd204071a70 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20aa00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd20415f140 con 0x7fd2041556f0 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20aa00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd20415f440 con 0x7fd2040ce2d0 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd209600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 0x7fd2040a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd209600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 0x7fd2040a9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33814/0 (socket says 172.21.15.163:33814) 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd209600640 1 -- 172.21.15.163:0/3531542305 learned_addr learned my addr 172.21.15.163:0/3531542305 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd20a000640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2041556f0 0x7fd2041e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.070+0000 7fd208c00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2040ce2d0 0x7fd2041e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd209600640 1 -- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2040ce2d0 msgr2=0x7fd2041e6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd209600640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2040ce2d0 0x7fd2041e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd209600640 1 -- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2041556f0 msgr2=0x7fd2041e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd209600640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2041556f0 0x7fd2041e6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd209600640 1 -- 172.21.15.163:0/3531542305 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd204140150 con 0x7fd204071a70 2024-07-26T06:14:14.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd209600640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 0x7fd2040a9740 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8034040 tx=0x7fd1f8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:14.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f803e030 con 0x7fd204071a70 2024-07-26T06:14:14.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1f8003e00 con 0x7fd204071a70 2024-07-26T06:14:14.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8035040 con 0x7fd204071a70 2024-07-26T06:14:14.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd204140430 con 0x7fd204071a70 2024-07-26T06:14:14.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.071+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2041409f0 con 0x7fd204071a70 2024-07-26T06:14:14.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.073+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20415f960 con 0x7fd204071a70 2024-07-26T06:14:14.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.074+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd1f8041030 con 0x7fd204071a70 2024-07-26T06:14:14.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.075+0000 7fd201600640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd1e40762d0 0x7fd1e4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.075+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd1f80b7e20 con 0x7fd204071a70 2024-07-26T06:14:14.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.076+0000 7fd208c00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd1e40762d0 0x7fd1e4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.077+0000 7fd208c00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd1e40762d0 0x7fd1e4078790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0005fd0 tx=0x7fd1f0005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:14.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.080+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1f8081770 con 0x7fd204071a70 2024-07-26T06:14:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.297+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2040b34f0 con 0x7fd204071a70 2024-07-26T06:14:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.300+0000 7fd201600640 1 -- 172.21.15.163:0/3531542305 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16878 (secure 0 0 0) 0x7fd1f8081070 con 0x7fd204071a70 2024-07-26T06:14:14.301 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:14.302 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":22,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:14.266263+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:13:32.708837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:13:33.488054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd1e40762d0 msgr2=0x7fd1e4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd1e40762d0 0x7fd1e4078790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0005fd0 tx=0x7fd1f0005850 comp rx=0 tx=0).stop 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 msgr2=0x7fd2040a9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 0x7fd2040a9740 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8034040 tx=0x7fd1f8003910 comp rx=0 tx=0).stop 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 shutdown_connections 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd1e40762d0 0x7fd1e4078790 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2041556f0 0x7fd2041e6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2040ce2d0 0x7fd2041e6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 --2- 172.21.15.163:0/3531542305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd204071a70 0x7fd2040a9740 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 >> 172.21.15.163:0/3531542305 conn(0x7fd2040b0e30 msgr2=0x7fd20415d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:14.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 shutdown_connections 2024-07-26T06:14:14.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.303+0000 7fd20aa00640 1 -- 172.21.15.163:0/3531542305 wait complete. 2024-07-26T06:14:14.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:14:14.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 -- 172.21.15.163:0/3764736561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1224005210 msgr2=0x7f122400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 --2- 172.21.15.163:0/3764736561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1224005210 0x7f122400b9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f123800c470 tx=0x7f1224005830 comp rx=0 tx=0).stop 2024-07-26T06:14:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 -- 172.21.15.163:0/3764736561 shutdown_connections 2024-07-26T06:14:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 --2- 172.21.15.163:0/3764736561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1224005210 0x7f122400b9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 -- 172.21.15.163:0/3764736561 >> 172.21.15.163:0/3764736561 conn(0x7f124006bf40 msgr2=0x7f124006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 -- 172.21.15.163:0/3764736561 shutdown_connections 2024-07-26T06:14:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.521+0000 7f1245a00640 1 -- 172.21.15.163:0/3764736561 wait complete. 2024-07-26T06:14:14.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.523+0000 7f1245a00640 1 Processor -- start 2024-07-26T06:14:14.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.523+0000 7f1245a00640 1 -- start start 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1224005210 0x7f1240146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1240156930 0x7f1240156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 0x7f12400c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f12400d1bd0 con 0x7f12400c61d0 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f12400d1a50 con 0x7f1224005210 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f12400d03b0 con 0x7f1240156930 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 0x7f12400c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.524+0000 7f1245000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 0x7f12400c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33844/0 (socket says 172.21.15.163:33844) 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 -- 172.21.15.163:0/137742209 learned_addr learned my addr 172.21.15.163:0/137742209 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f123fe00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1224005210 0x7f1240146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f123f400640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1240156930 0x7f1240156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 -- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1240156930 msgr2=0x7f1240156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1240156930 0x7f1240156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 -- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1224005210 msgr2=0x7f1240146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1224005210 0x7f1240146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 -- 172.21.15.163:0/137742209 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1240153510 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1245000640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 0x7f12400c6680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f123800c590 tx=0x7f1238006ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.525+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238039070 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.526+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f123802b620 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.526+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123802ba50 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.526+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12400c6de0 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.526+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12401473a0 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.527+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12400d0880 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.528+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f123802bbf0 con 0x7f12400c61d0 2024-07-26T06:14:14.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.530+0000 7f1237e00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f121c076300 0x7f121c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:14.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.530+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f12380b6fa0 con 0x7f12400c61d0 2024-07-26T06:14:14.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.530+0000 7f123fe00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f121c076300 0x7f121c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:14.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.531+0000 7f123fe00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f121c076300 0x7f121c0787c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1230005fd0 tx=0x7f12300093c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:14.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.534+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1240153510 con 0x7f12400c61d0 2024-07-26T06:14:14.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.782+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f12400b3b30 con 0x7f12400c61d0 2024-07-26T06:14:14.785 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:14:14.784+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:14:14.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.787+0000 7f1237e00640 1 -- 172.21.15.163:0/137742209 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f12400735e0 con 0x7f12400c61d0 2024-07-26T06:14:14.788 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:14:14.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f121c076300 msgr2=0x7f121c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f121c076300 0x7f121c0787c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1230005fd0 tx=0x7f12300093c0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 msgr2=0x7f12400c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 0x7f12400c6680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f123800c590 tx=0x7f1238006ee0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 shutdown_connections 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f121c076300 0x7f121c0787c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12400c61d0 0x7f12400c6680 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1240156930 0x7f1240156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 --2- 172.21.15.163:0/137742209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1224005210 0x7f1240146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.790+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 >> 172.21.15.163:0/137742209 conn(0x7f124006bf40 msgr2=0x7f124015be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:14.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.791+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 shutdown_connections 2024-07-26T06:14:14.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.791+0000 7f1245a00640 1 -- 172.21.15.163:0/137742209 wait complete. 2024-07-26T06:14:14.806 INFO:tasks.ceph.mds.c.smithi163.stderr: -5469> 2024-07-26T06:13:39.991+0000 7f7e23800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:14:14.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:14:14.863 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:14:14.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 -- 172.21.15.163:0/3281527066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e300cdd90 msgr2=0x7f1e300ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:14.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3281527066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e300cdd90 0x7f1e300ce170 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e20005fd0 tx=0x7f1e2002a5d0 comp rx=0 tx=0).stop 2024-07-26T06:14:14.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 -- 172.21.15.163:0/3281527066 shutdown_connections 2024-07-26T06:14:15.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3281527066 >> v2:172.21.15.164:3300/0 conn(0x7f1e301413b0 0x7f1e3015b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3281527066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e300cdd90 0x7f1e300ce170 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 -- 172.21.15.163:0/3281527066 >> 172.21.15.163:0/3281527066 conn(0x7f1e3006bf40 msgr2=0x7f1e3006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:15.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 -- 172.21.15.163:0/3281527066 shutdown_connections 2024-07-26T06:14:15.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:14.998+0000 7f1e36c00640 1 -- 172.21.15.163:0/3281527066 wait complete. 2024-07-26T06:14:15.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.000+0000 7f1e36c00640 1 Processor -- start 2024-07-26T06:14:15.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.001+0000 7f1e36c00640 1 -- start start 2024-07-26T06:14:15.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e36c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1e30074020 0x7f1e3015b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e36c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 0x7f1e3013d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e36c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e301413b0 0x7f1e3015d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e36c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3013ef70 con 0x7f1e300cdd90 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e36c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3013edf0 con 0x7f1e30074020 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e36c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1e300744d0 con 0x7f1e301413b0 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 0x7f1e3013d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 0x7f1e3013d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37274/0 (socket says 172.21.15.163:37274) 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 -- 172.21.15.163:0/3189961173 learned_addr learned my addr 172.21.15.163:0/3189961173 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e2fe00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1e30074020 0x7f1e3015b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e34c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e301413b0 0x7f1e3015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 -- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e301413b0 msgr2=0x7f1e3015d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e301413b0 0x7f1e3015d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 -- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1e30074020 msgr2=0x7f1e3015b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.002+0000 7f1e27e00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1e30074020 0x7f1e3015b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e27e00640 1 -- 172.21.15.163:0/3189961173 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3015b8d0 con 0x7f1e300cdd90 2024-07-26T06:14:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e27e00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 0x7f1e3013d8c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800e910 tx=0x7f1e1800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800cd80 con 0x7f1e300cdd90 2024-07-26T06:14:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e18010430 con 0x7f1e300cdd90 2024-07-26T06:14:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e301f7450 con 0x7f1e300cdd90 2024-07-26T06:14:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e18010860 con 0x7f1e300cdd90 2024-07-26T06:14:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.003+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e301f7a10 con 0x7f1e300cdd90 2024-07-26T06:14:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.005+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e30069810 con 0x7f1e300cdd90 2024-07-26T06:14:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.006+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1e18010a00 con 0x7f1e300cdd90 2024-07-26T06:14:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.006+0000 7f1e2cc00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1e0c0762d0 0x7f1e0c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.007+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7f1e180983f0 con 0x7f1e300cdd90 2024-07-26T06:14:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.007+0000 7f1e2fe00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1e0c0762d0 0x7f1e0c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.008+0000 7f1e2fe00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1e0c0762d0 0x7f1e0c078790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1e20033000 tx=0x7f1e20004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.011+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e18054af0 con 0x7f1e300cdd90 2024-07-26T06:14:15.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.271+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1e300ce170 con 0x7f1e300cdd90 2024-07-26T06:14:15.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.791+0000 7f1e2cc00640 1 -- 172.21.15.163:0/3189961173 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f1e18061b60 con 0x7f1e300cdd90 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1e0c0762d0 msgr2=0x7f1e0c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1e0c0762d0 0x7f1e0c078790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1e20033000 tx=0x7f1e20004730 comp rx=0 tx=0).stop 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 msgr2=0x7f1e3013d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 0x7f1e3013d8c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800e910 tx=0x7f1e1800edd0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 shutdown_connections 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1e0c0762d0 0x7f1e0c078790 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1e301413b0 0x7f1e3015d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e300cdd90 0x7f1e3013d8c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 --2- 172.21.15.163:0/3189961173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1e30074020 0x7f1e3015b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:15.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 >> 172.21.15.163:0/3189961173 conn(0x7f1e3006bf40 msgr2=0x7f1e30140400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:15.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 shutdown_connections 2024-07-26T06:14:15.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:15.795+0000 7f1e36c00640 1 -- 172.21.15.163:0/3189961173 wait complete. 2024-07-26T06:14:15.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:14:16.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.026+0000 7f5a61a00640 1 -- 172.21.15.163:0/1735422109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a40004d40 msgr2=0x7f5a400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:16.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.026+0000 7f5a61a00640 1 --2- 172.21.15.163:0/1735422109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a40004d40 0x7f5a400051c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5a54005fd0 tx=0x7f5a40005830 comp rx=0 tx=0).stop 2024-07-26T06:14:16.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.026+0000 7f5a5b400640 1 -- 172.21.15.163:0/1735422109 reap_dead start 2024-07-26T06:14:16.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.027+0000 7f5a61a00640 1 -- 172.21.15.163:0/1735422109 shutdown_connections 2024-07-26T06:14:16.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.027+0000 7f5a61a00640 1 -- 172.21.15.163:0/1735422109 >> 172.21.15.163:0/1735422109 conn(0x7f5a5c06d420 msgr2=0x7f5a5c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:16.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.027+0000 7f5a61a00640 1 -- 172.21.15.163:0/1735422109 shutdown_connections 2024-07-26T06:14:16.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.027+0000 7f5a61a00640 1 -- 172.21.15.163:0/1735422109 wait complete. 2024-07-26T06:14:16.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.028+0000 7f5a61a00640 1 Processor -- start 2024-07-26T06:14:16.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.029+0000 7f5a61a00640 1 -- start start 2024-07-26T06:14:16.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a61a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a40004d40 0x7f5a5c1f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:16.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a61a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 0x7f5a5c0add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a61a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5c150b00 0x7f5a5c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a61a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c15f0b0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a61a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c15ef30 con 0x7f5a40004d40 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a61a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5c15c690 con 0x7f5a5c150b00 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 0x7f5a5c0add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 0x7f5a5c0add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37292/0 (socket says 172.21.15.163:37292) 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 -- 172.21.15.163:0/80106427 learned_addr learned my addr 172.21.15.163:0/80106427 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a5aa00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a40004d40 0x7f5a5c1f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a5b400640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5c150b00 0x7f5a5c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 -- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5c150b00 msgr2=0x7f5a5c150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5c150b00 0x7f5a5c150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 -- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a40004d40 msgr2=0x7f5a5c1f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a40004d40 0x7f5a5c1f6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.030+0000 7f5a53e00640 1 -- 172.21.15.163:0/80106427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5c1f7040 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.031+0000 7f5a53e00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 0x7f5a5c0add60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400c500 tx=0x7f5a4400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:16.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.031+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44016020 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.031+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a440053e0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.031+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5c1517e0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.031+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44007280 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.031+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5c151da0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.032+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a5c0ae2a0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.033+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5a44007420 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.034+0000 7f5a52a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5a380762d0 0x7f5a38078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:16.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.035+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7f5a44096f00 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.035+0000 7f5a5aa00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5a380762d0 0x7f5a38078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:16.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.036+0000 7f5a5aa00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5a380762d0 0x7f5a38078790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c001eb0 tx=0x7f5a4c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:16.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.039+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5a440607c0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.254+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5a5c06bf40 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.794+0000 7f5a52a00640 1 -- 172.21.15.163:0/80106427 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7f5a440600c0 con 0x7f5a5c0ad8e0 2024-07-26T06:14:16.796 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:14:16.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5a380762d0 msgr2=0x7f5a38078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:16.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5a380762d0 0x7f5a38078790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c001eb0 tx=0x7f5a4c009040 comp rx=0 tx=0).stop 2024-07-26T06:14:16.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 msgr2=0x7f5a5c0add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 0x7f5a5c0add60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400c500 tx=0x7f5a4400c9c0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 shutdown_connections 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5a380762d0 0x7f5a38078790 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5c150b00 0x7f5a5c150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a5c0ad8e0 0x7f5a5c0add60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 --2- 172.21.15.163:0/80106427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a40004d40 0x7f5a5c1f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.798+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 >> 172.21.15.163:0/80106427 conn(0x7f5a5c06d420 msgr2=0x7f5a5c0cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.799+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 shutdown_connections 2024-07-26T06:14:16.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:16.799+0000 7f5a61a00640 1 -- 172.21.15.163:0/80106427 wait complete. 2024-07-26T06:14:16.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:14:17.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe959400640 1 -- 172.21.15.163:0/955325349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 msgr2=0x7fe938003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:17.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe959400640 1 --2- 172.21.15.163:0/955325349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe938003e20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c4a0 tx=0x7fe938005830 comp rx=0 tx=0).stop 2024-07-26T06:14:17.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe958a00640 1 -- 172.21.15.163:0/955325349 reap_dead start 2024-07-26T06:14:17.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe959400640 1 -- 172.21.15.163:0/955325349 shutdown_connections 2024-07-26T06:14:17.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe959400640 1 -- 172.21.15.163:0/955325349 >> 172.21.15.163:0/955325349 conn(0x7fe9540cfda0 msgr2=0x7fe9540d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:17.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe959400640 1 -- 172.21.15.163:0/955325349 shutdown_connections 2024-07-26T06:14:17.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.030+0000 7fe959400640 1 -- 172.21.15.163:0/955325349 wait complete. 2024-07-26T06:14:17.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.032+0000 7fe959400640 1 Processor -- start 2024-07-26T06:14:17.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.032+0000 7fe959400640 1 -- start start 2024-07-26T06:14:17.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe959400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe95415bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:17.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe959400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9541559a0 0x7fe954155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:17.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe959400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9540ca3f0 0x7fe9540ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:17.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe959400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540b6b80 con 0x7fe938003a00 2024-07-26T06:14:17.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe959400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540b6a00 con 0x7fe9541559a0 2024-07-26T06:14:17.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe959400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe9540b6d00 con 0x7fe9540ca3f0 2024-07-26T06:14:17.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe953e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe95415bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:17.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe953e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe95415bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37316/0 (socket says 172.21.15.163:37316) 2024-07-26T06:14:17.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.033+0000 7fe953e00640 1 -- 172.21.15.163:0/3626829401 learned_addr learned my addr 172.21.15.163:0/3626829401 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:17.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe958a00640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9540ca3f0 0x7fe9540ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:17.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe953e00640 1 -- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9540ca3f0 msgr2=0x7fe9540ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:17.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe953e00640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9540ca3f0 0x7fe9540ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe953e00640 1 -- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9541559a0 msgr2=0x7fe954155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:17.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe953e00640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9541559a0 0x7fe954155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe953e00640 1 -- 172.21.15.163:0/3626829401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe954153500 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe953e00640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe95415bd30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c590 tx=0x7fe9440012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9440065f0 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe94402b660 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94402ba70 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.034+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe954156340 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.035+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9541f73e0 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.036+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9540b64c0 con 0x7fe938003a00 2024-07-26T06:14:17.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.037+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe94402bc10 con 0x7fe938003a00 2024-07-26T06:14:17.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.038+0000 7fe94be00640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe930076300 0x7fe9300787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:17.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.038+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5473+0+0 (secure 0 0 0) 0x7fe9440b75f0 con 0x7fe938003a00 2024-07-26T06:14:17.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.038+0000 7fe953400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe930076300 0x7fe9300787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:17.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.039+0000 7fe953400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe930076300 0x7fe9300787c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe93c005ba0 tx=0x7fe93c005a70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:17.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.042+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe9540b6f40 con 0x7fe938003a00 2024-07-26T06:14:17.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.247+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe9540ce650 con 0x7fe938003a00 2024-07-26T06:14:17.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.798+0000 7fe94be00640 1 -- 172.21.15.163:0/3626829401 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7fe944080f00 con 0x7fe938003a00 2024-07-26T06:14:17.800 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:14:17.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.803+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe930076300 msgr2=0x7fe9300787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:17.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.803+0000 7fe959400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe930076300 0x7fe9300787c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe93c005ba0 tx=0x7fe93c005a70 comp rx=0 tx=0).stop 2024-07-26T06:14:17.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.803+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 msgr2=0x7fe95415bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:17.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.803+0000 7fe959400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe95415bd30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe94400c590 tx=0x7fe9440012d0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 shutdown_connections 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe930076300 0x7fe9300787c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9540ca3f0 0x7fe9540ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9541559a0 0x7fe954155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 --2- 172.21.15.163:0/3626829401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe938003a00 0x7fe95415bd30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 >> 172.21.15.163:0/3626829401 conn(0x7fe9540cfda0 msgr2=0x7fe9540b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.804+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 shutdown_connections 2024-07-26T06:14:17.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:17.805+0000 7fe959400640 1 -- 172.21.15.163:0/3626829401 wait complete. 2024-07-26T06:14:17.828 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:14:17.828 INFO:tasks.ceph.mds.a:Stopping old one... 2024-07-26T06:14:17.828 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:14:17.828 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:14:17.828 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:14:17.828 INFO:tasks.ceph.mds.c:Stopping old one... 2024-07-26T06:14:17.829 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:14:17.829 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:14:17.829 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:14:17.829 INFO:tasks.ceph.mds.b:Stopping old one... 2024-07-26T06:14:17.829 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:14:17.829 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:14:17.829 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:14:17.830 INFO:tasks.ceph.mds.d:Stopping old one... 2024-07-26T06:14:17.830 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:14:17.830 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:14:17.849 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:14:17.849 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:14:17.850 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:14:17.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:14:17.852 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:14:17.852 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:14:17.854 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:14:17.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:14:17.898 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:14:17.899 DEBUG:teuthology.parallel:result is None 2024-07-26T06:14:17.900 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:14:17.900 DEBUG:teuthology.parallel:result is None 2024-07-26T06:14:17.904 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:14:17.905 DEBUG:teuthology.parallel:result is None 2024-07-26T06:14:17.906 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:14:17.906 DEBUG:teuthology.parallel:result is None 2024-07-26T06:14:17.906 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-07-26T06:14:18.047 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:14:18.051 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:14:18.051 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:14:18.051 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:14:18.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 -- 172.21.15.163:0/944978446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b400b5800 msgr2=0x7f3b400b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:18.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 --2- 172.21.15.163:0/944978446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b400b5800 0x7f3b400b6e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009be0 tx=0x7f3b3c02a9b0 comp rx=0 tx=0).stop 2024-07-26T06:14:18.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 -- 172.21.15.163:0/944978446 shutdown_connections 2024-07-26T06:14:18.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 --2- 172.21.15.163:0/944978446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b400b5800 0x7f3b400b6e30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:18.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 -- 172.21.15.163:0/944978446 >> 172.21.15.163:0/944978446 conn(0x7f3b400d36b0 msgr2=0x7f3b400d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:18.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 -- 172.21.15.163:0/944978446 shutdown_connections 2024-07-26T06:14:18.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.089+0000 7f3b48800640 1 -- 172.21.15.163:0/944978446 wait complete. 2024-07-26T06:14:18.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.091+0000 7f3b48800640 1 Processor -- start 2024-07-26T06:14:18.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.091+0000 7f3b48800640 1 -- start start 2024-07-26T06:14:18.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b48800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 0x7f3b4014ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:18.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b48800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b400b5800 0x7f3b401590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:18.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b48800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4015d650 0x7f3b400b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b48800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3b400b7370 con 0x7f3b400b53b0 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b48800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3b400b74f0 con 0x7f3b400b5800 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b48800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3b400b7670 con 0x7f3b4015d650 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.092+0000 7f3b45e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 0x7f3b4014ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 0x7f3b4014ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37332/0 (socket says 172.21.15.163:37332) 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 -- 172.21.15.163:0/4011401862 learned_addr learned my addr 172.21.15.163:0/4011401862 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45400640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b400b5800 0x7f3b401590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:18.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b46800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4015d650 0x7f3b400b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:18.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 -- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4015d650 msgr2=0x7f3b400b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:18.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4015d650 0x7f3b400b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 -- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b400b5800 msgr2=0x7f3b401590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b400b5800 0x7f3b401590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 -- 172.21.15.163:0/4011401862 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b40159b40 con 0x7f3b400b53b0 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b45e00640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 0x7f3b4014ba10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c730 tx=0x7f3b3400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.093+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34017070 con 0x7f3b400b53b0 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.094+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b34006890 con 0x7f3b400b53b0 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.094+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34006d20 con 0x7f3b400b53b0 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.094+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b40159e20 con 0x7f3b400b53b0 2024-07-26T06:14:18.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.094+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b4015a3e0 con 0x7f3b400b53b0 2024-07-26T06:14:18.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.095+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b400d1990 con 0x7f3b400b53b0 2024-07-26T06:14:18.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.096+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3b3401a070 con 0x7f3b400b53b0 2024-07-26T06:14:18.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.097+0000 7f3b3a000640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3b24076300 0x7f3b240787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:18.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.097+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7f3b340968c0 con 0x7f3b400b53b0 2024-07-26T06:14:18.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.098+0000 7f3b45400640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3b24076300 0x7f3b240787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:18.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.098+0000 7f3b45400640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3b24076300 0x7f3b240787c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3b30000c00 tx=0x7f3b30009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:18.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.101+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3b400ca3a0 con 0x7f3b400b53b0 2024-07-26T06:14:18.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:18.326+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f3b400ca180 con 0x7f3b400b53b0 2024-07-26T06:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.327+0000 7f3b3a000640 1 -- 172.21.15.163:0/4011401862 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) ==== 97+0+0 (secure 0 0 0) 0x7f3b3405fec0 con 0x7f3b400b53b0 2024-07-26T06:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr: removed all blocklist entries 2024-07-26T06:14:19.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3b24076300 msgr2=0x7f3b240787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:19.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3b24076300 0x7f3b240787c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3b30000c00 tx=0x7f3b30009040 comp rx=0 tx=0).stop 2024-07-26T06:14:19.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 msgr2=0x7f3b4014ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:19.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 0x7f3b4014ba10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c730 tx=0x7f3b3400cbf0 comp rx=0 tx=0).stop 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 shutdown_connections 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3b24076300 0x7f3b240787c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4015d650 0x7f3b400b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b400b5800 0x7f3b401590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 --2- 172.21.15.163:0/4011401862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b400b53b0 0x7f3b4014ba10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.330+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 >> 172.21.15.163:0/4011401862 conn(0x7f3b400d36b0 msgr2=0x7f3b400d3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.331+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 shutdown_connections 2024-07-26T06:14:19.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.331+0000 7f3b48800640 1 -- 172.21.15.163:0/4011401862 wait complete. 2024-07-26T06:14:19.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-07-26T06:14:19.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 -- 172.21.15.163:0/2413148090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280cbad0 msgr2=0x7fbb280cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:19.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2413148090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280cbad0 0x7fbb280cd530 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009830 tx=0x7fbb1402a510 comp rx=0 tx=0).stop 2024-07-26T06:14:19.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 -- 172.21.15.163:0/2413148090 shutdown_connections 2024-07-26T06:14:19.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2413148090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280cbad0 0x7fbb280cd530 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2413148090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb28159370 0x7fbb28159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 -- 172.21.15.163:0/2413148090 >> 172.21.15.163:0/2413148090 conn(0x7fbb280b0e30 msgr2=0x7fbb280b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:24.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 -- 172.21.15.163:0/2413148090 shutdown_connections 2024-07-26T06:14:24.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.553+0000 7fbb2e000640 1 -- 172.21.15.163:0/2413148090 wait complete. 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.554+0000 7fbb2e000640 1 Processor -- start 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.555+0000 7fbb2e000640 1 -- start start 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.555+0000 7fbb2e000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb280b4420 0x7fbb28157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.555+0000 7fbb2e000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb28159370 0x7fbb28150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 0x7fbb280719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2e000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb280cda70 con 0x7fbb280afc40 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2e000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2806f500 con 0x7fbb28159370 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2e000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbb280cdbf0 con 0x7fbb280b4420 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 0x7fbb280719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 0x7fbb280719b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37342/0 (socket says 172.21.15.163:37342) 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 -- 172.21.15.163:0/2360169228 learned_addr learned my addr 172.21.15.163:0/2360169228 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2cc00640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb280b4420 0x7fbb28157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 -- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb280b4420 msgr2=0x7fbb28157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb280b4420 0x7fbb28157370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 -- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb28159370 msgr2=0x7fbb28150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb28159370 0x7fbb28150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.556+0000 7fbb2d600640 1 -- 172.21.15.163:0/2360169228 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb28157f10 con 0x7fbb280afc40 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.557+0000 7fbb2d600640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 0x7fbb280719b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000cbf0 tx=0x7fbb2000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.557+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20016020 con 0x7fbb280afc40 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.557+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb2000ae00 con 0x7fbb280afc40 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.557+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20012680 con 0x7fbb280afc40 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.557+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb28157c30 con 0x7fbb280afc40 2024-07-26T06:14:24.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.557+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb28073e20 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.558+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb280739a0 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.559+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb20019020 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.560+0000 7fbb24c00640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbb080762d0 0x7fbb08078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.560+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fbb20097c10 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.561+0000 7fbb2cc00640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbb080762d0 0x7fbb08078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.562+0000 7fbb2cc00640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbb080762d0 0x7fbb08078790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0058c0 tx=0x7fbb1c005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.564+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb20020020 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.852+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fbb280cd530 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.853+0000 7fbb24c00640 1 -- 172.21.15.163:0/2360169228 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v3) ==== 80+0+6645 (secure 0 0 0) 0x7fbb20061820 con 0x7fbb280afc40 2024-07-26T06:14:24.035 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_dump": [ 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.a", 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA==", 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.036 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.b", 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg==", 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.037 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.c", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw==", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.d", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg==", 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.038 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.0", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA==", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.1", 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA==", 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.2", 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA==", 2024-07-26T06:14:24.040 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.3", 2024-07-26T06:14:24.041 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg==", 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.4", 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA==", 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.042 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.5", 2024-07-26T06:14:24.043 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g==", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.6", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ==", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.044 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.7", 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA==", 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.0", 2024-07-26T06:14:24.045 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew==", 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.1", 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ==", 2024-07-26T06:14:24.046 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.047 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.047 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:14:24.047 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:14:24.047 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:14:24.047 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.048 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.2", 2024-07-26T06:14:24.048 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw==", 2024-07-26T06:14:24.048 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.049 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.049 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:14:24.049 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:14:24.049 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:14:24.049 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.3", 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw==", 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:14:24.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.admin", 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm+ibKBxAA1ZBGvVy5rNIzMWwOFwFJFw==", 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.051 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.bootstrap-mds", 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQB0PqNmmnwNDhAAH+Y/flsrjTHcI38PVItd9g==", 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile bootstrap-mds" 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.bootstrap-mgr", 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQB0PqNmr4wNDhAAxPFZSyuw0BlUpqq80E9sRA==", 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.052 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile bootstrap-mgr" 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.bootstrap-osd", 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQB0PqNmap0NDhAAbNgvYGGAvz6ybL+GHTeF5g==", 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile bootstrap-osd" 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.bootstrap-rbd", 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQB0PqNmL6wNDhAAp3T/neXnNHdygh1QngsHWA==", 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile bootstrap-rbd" 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQB0PqNmYbkNDhAAgKlCPpYdd8gMcKqhdcZtiw==", 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.bootstrap-rgw", 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQB0PqNmo8INDhAAGp9JGOwctmKZxoPphLtnTg==", 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile bootstrap-rgw" 2024-07-26T06:14:24.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.x", 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ==", 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.y", 2024-07-26T06:14:24.056 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ==", 2024-07-26T06:14:24.057 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:14:24.057 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:14:24.057 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:14:24.057 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:14:24.058 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.058 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:24.058 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:14:24.058 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:14:24.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbb080762d0 msgr2=0x7fbb08078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbb080762d0 0x7fbb08078790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0058c0 tx=0x7fbb1c005850 comp rx=0 tx=0).stop 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 msgr2=0x7fbb280719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 0x7fbb280719b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000cbf0 tx=0x7fbb2000a430 comp rx=0 tx=0).stop 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 shutdown_connections 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbb080762d0 0x7fbb08078790 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb280afc40 0x7fbb280719b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb28159370 0x7fbb28150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 --2- 172.21.15.163:0/2360169228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb280b4420 0x7fbb28157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 >> 172.21.15.163:0/2360169228 conn(0x7fbb280b0e30 msgr2=0x7fbb28152ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:24.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 shutdown_connections 2024-07-26T06:14:24.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:19.856+0000 7fbb2e000640 1 -- 172.21.15.163:0/2360169228 wait complete. 2024-07-26T06:14:24.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-07-26T06:14:24.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75c00640 1 -- 172.21.15.163:0/4272277986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda50004d40 msgr2=0x7fda500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75c00640 1 --2- 172.21.15.163:0/4272277986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda50004d40 0x7fda500051c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fda6800c2e0 tx=0x7fda50005830 comp rx=0 tx=0).stop 2024-07-26T06:14:24.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75200640 1 -- 172.21.15.163:0/4272277986 reap_dead start 2024-07-26T06:14:24.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75c00640 1 -- 172.21.15.163:0/4272277986 shutdown_connections 2024-07-26T06:14:24.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75c00640 1 -- 172.21.15.163:0/4272277986 >> 172.21.15.163:0/4272277986 conn(0x7fda700b0e30 msgr2=0x7fda700b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:24.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75c00640 1 -- 172.21.15.163:0/4272277986 shutdown_connections 2024-07-26T06:14:24.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.270+0000 7fda75c00640 1 -- 172.21.15.163:0/4272277986 wait complete. 2024-07-26T06:14:24.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.272+0000 7fda75c00640 1 Processor -- start 2024-07-26T06:14:24.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.272+0000 7fda75c00640 1 -- start start 2024-07-26T06:14:24.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda50004d40 0x7fda7015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda701f0ca0 0x7fda701f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 0x7fda70073790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fda700cb870 con 0x7fda701f2960 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fda70073300 con 0x7fda50004d40 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fda700cb9f0 con 0x7fda701f0ca0 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.273+0000 7fda75200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 0x7fda70073790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 0x7fda70073790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37356/0 (socket says 172.21.15.163:37356) 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 -- 172.21.15.163:0/157700642 learned_addr learned my addr 172.21.15.163:0/157700642 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda6fe00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda50004d40 0x7fda7015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda6f400640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda701f0ca0 0x7fda701f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 -- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda701f0ca0 msgr2=0x7fda701f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda701f0ca0 0x7fda701f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 -- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda50004d40 msgr2=0x7fda7015b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda50004d40 0x7fda7015b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 -- 172.21.15.163:0/157700642 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7015bba0 con 0x7fda701f2960 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda6fe00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda50004d40 0x7fda7015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda75200640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 0x7fda70073790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fda6802b130 tx=0x7fda68006630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.274+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda68031070 con 0x7fda701f2960 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.275+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda68006c50 con 0x7fda701f2960 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.275+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6802c8e0 con 0x7fda701f2960 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.275+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7015be80 con 0x7fda701f2960 2024-07-26T06:14:24.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.275+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda70156830 con 0x7fda701f2960 2024-07-26T06:14:24.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.276+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda7015c3a0 con 0x7fda701f2960 2024-07-26T06:14:24.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.277+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fda68034070 con 0x7fda701f2960 2024-07-26T06:14:24.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.278+0000 7fda67e00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fda4c076300 0x7fda4c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.278+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fda680b6ae0 con 0x7fda701f2960 2024-07-26T06:14:24.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.279+0000 7fda6fe00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fda4c076300 0x7fda4c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.280+0000 7fda6fe00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fda4c076300 0x7fda4c0787c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fda70152620 tx=0x7fda60005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:24.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.283+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda7015c3a0 con 0x7fda701f2960 2024-07-26T06:14:24.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.554+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fda700b3fb0 con 0x7fda701f2960 2024-07-26T06:14:24.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.559+0000 7fda67e00640 1 -- 172.21.15.163:0/157700642 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v4) ==== 90+0+0 (secure 0 0 0) 0x7fda680806f0 con 0x7fda701f2960 2024-07-26T06:14:24.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fda4c076300 msgr2=0x7fda4c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fda4c076300 0x7fda4c0787c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fda70152620 tx=0x7fda60005f90 comp rx=0 tx=0).stop 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 msgr2=0x7fda70073790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 0x7fda70073790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fda6802b130 tx=0x7fda68006630 comp rx=0 tx=0).stop 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 shutdown_connections 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fda4c076300 0x7fda4c0787c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda701f2960 0x7fda70073790 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda701f0ca0 0x7fda701f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 --2- 172.21.15.163:0/157700642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda50004d40 0x7fda7015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.562+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 >> 172.21.15.163:0/157700642 conn(0x7fda700b0e30 msgr2=0x7fda700af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.563+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 shutdown_connections 2024-07-26T06:14:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.563+0000 7fda75c00640 1 -- 172.21.15.163:0/157700642 wait complete. 2024-07-26T06:14:24.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-07-26T06:14:24.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 -- 172.21.15.163:0/1247106109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 msgr2=0x7fe47c073cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 --2- 172.21.15.163:0/1247106109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c073cd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe474009900 tx=0x7fe47402a4d0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 -- 172.21.15.163:0/1247106109 shutdown_connections 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 --2- 172.21.15.163:0/1247106109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c073cd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 --2- 172.21.15.163:0/1247106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe47c0b3bd0 0x7fe47c0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 -- 172.21.15.163:0/1247106109 >> 172.21.15.163:0/1247106109 conn(0x7fe47c0b0e30 msgr2=0x7fe47c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 -- 172.21.15.163:0/1247106109 shutdown_connections 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.782+0000 7fe481600640 1 -- 172.21.15.163:0/1247106109 wait complete. 2024-07-26T06:14:24.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.784+0000 7fe481600640 1 Processor -- start 2024-07-26T06:14:24.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.784+0000 7fe481600640 1 -- start start 2024-07-26T06:14:24.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe481600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe481600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe47c0ae870 0x7fe47c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe481600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe47c0b3bd0 0x7fe47c146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe481600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c073380 con 0x7fe47c073820 2024-07-26T06:14:24.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe481600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c15a1d0 con 0x7fe47c0ae870 2024-07-26T06:14:24.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe481600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe47c15a350 con 0x7fe47c0b3bd0 2024-07-26T06:14:24.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.785+0000 7fe47be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37372/0 (socket says 172.21.15.163:37372) 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 -- 172.21.15.163:0/1650786349 learned_addr learned my addr 172.21.15.163:0/1650786349 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47b400640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe47c0ae870 0x7fe47c1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe480c00640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe47c0b3bd0 0x7fe47c146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 -- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe47c0b3bd0 msgr2=0x7fe47c146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe47c0b3bd0 0x7fe47c146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 -- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe47c0ae870 msgr2=0x7fe47c1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe47c0ae870 0x7fe47c1577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 -- 172.21.15.163:0/1650786349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c158010 con 0x7fe47c073820 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe47be00640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c159430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00ba50 tx=0x7fe46c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.786+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c019070 con 0x7fe47c073820 2024-07-26T06:14:24.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.787+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47c1582f0 con 0x7fe47c073820 2024-07-26T06:14:24.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.787+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe46c00ccf0 con 0x7fe47c073820 2024-07-26T06:14:24.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.787+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c0146a0 con 0x7fe47c073820 2024-07-26T06:14:24.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.787+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47c14d990 con 0x7fe47c073820 2024-07-26T06:14:24.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.788+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47c14d7a0 con 0x7fe47c073820 2024-07-26T06:14:24.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.789+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe46c01c070 con 0x7fe47c073820 2024-07-26T06:14:24.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.790+0000 7fe473e00640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4580762d0 0x7fe458078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:24.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.790+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fe46c097310 con 0x7fe47c073820 2024-07-26T06:14:24.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.790+0000 7fe47b400640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4580762d0 0x7fe458078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:24.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.791+0000 7fe47b400640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4580762d0 0x7fe458078790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe47c159970 tx=0x7fe464009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:24.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:24.794+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe46c060f30 con 0x7fe47c073820 2024-07-26T06:14:25.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.068+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fe47c0745c0 con 0x7fe47c073820 2024-07-26T06:14:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.074+0000 7fe473e00640 1 -- 172.21.15.163:0/1650786349 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7fe46c0970f0 con 0x7fe47c073820 2024-07-26T06:14:25.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4580762d0 msgr2=0x7fe458078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4580762d0 0x7fe458078790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe47c159970 tx=0x7fe464009040 comp rx=0 tx=0).stop 2024-07-26T06:14:25.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 msgr2=0x7fe47c159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c159430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00ba50 tx=0x7fe46c00bf10 comp rx=0 tx=0).stop 2024-07-26T06:14:25.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 shutdown_connections 2024-07-26T06:14:25.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4580762d0 0x7fe458078790 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe47c0b3bd0 0x7fe47c146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe47c0ae870 0x7fe47c1577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 --2- 172.21.15.163:0/1650786349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe47c073820 0x7fe47c159430 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 >> 172.21.15.163:0/1650786349 conn(0x7fe47c0b0e30 msgr2=0x7fe47c148e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:25.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.076+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 shutdown_connections 2024-07-26T06:14:25.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.077+0000 7fe481600640 1 -- 172.21.15.163:0/1650786349 wait complete. 2024-07-26T06:14:25.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-07-26T06:14:25.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 -- 172.21.15.163:0/3632719368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f780ce4a0 msgr2=0x7f5f78071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 --2- 172.21.15.163:0/3632719368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f780ce4a0 0x7f5f78071530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5f780b1e00 tx=0x7f5f6402a560 comp rx=0 tx=0).stop 2024-07-26T06:14:25.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 -- 172.21.15.163:0/3632719368 shutdown_connections 2024-07-26T06:14:25.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 --2- 172.21.15.163:0/3632719368 >> v2:172.21.15.164:3300/0 conn(0x7f5f78071a70 0x7f5f780b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 --2- 172.21.15.163:0/3632719368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f780ce4a0 0x7f5f78071530 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 -- 172.21.15.163:0/3632719368 >> 172.21.15.163:0/3632719368 conn(0x7f5f780b0e30 msgr2=0x7f5f780b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:25.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 -- 172.21.15.163:0/3632719368 shutdown_connections 2024-07-26T06:14:25.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.297+0000 7f5f77e00640 1 -- 172.21.15.163:0/3632719368 wait complete. 2024-07-26T06:14:25.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.299+0000 7f5f77e00640 1 Processor -- start 2024-07-26T06:14:25.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.299+0000 7f5f77e00640 1 -- start start 2024-07-26T06:14:25.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.300+0000 7f5f77e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78071a70 0x7f5f78143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780ce4a0 0x7f5f781442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 0x7f5f781e46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5f780b3fe0 con 0x7f5f781556f0 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5f780b3e60 con 0x7f5f780ce4a0 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5f780b4160 con 0x7f5f78071a70 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 0x7f5f781e46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 0x7f5f781e46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49072/0 (socket says 172.21.15.163:49072) 2024-07-26T06:14:25.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 -- 172.21.15.163:0/2981569226 learned_addr learned my addr 172.21.15.163:0/2981569226 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:25.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f76000640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780ce4a0 0x7f5f781442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f76a00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78071a70 0x7f5f78143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 -- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78071a70 msgr2=0x7f5f78143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78071a70 0x7f5f78143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 -- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780ce4a0 msgr2=0x7f5f781442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780ce4a0 0x7f5f781442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f77400640 1 -- 172.21.15.163:0/2981569226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f781471c0 con 0x7f5f781556f0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.301+0000 7f5f76000640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780ce4a0 0x7f5f781442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.302+0000 7f5f77400640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 0x7f5f781e46a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000cad0 tx=0x7f5f7000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.302+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000ae20 con 0x7f5f781556f0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.302+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f70004b20 con 0x7f5f781556f0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.302+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70005020 con 0x7f5f781556f0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.302+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f78156c90 con 0x7f5f781556f0 2024-07-26T06:14:25.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.302+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f78157250 con 0x7f5f781556f0 2024-07-26T06:14:25.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.303+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f780af310 con 0x7f5f781556f0 2024-07-26T06:14:25.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.304+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5f700051c0 con 0x7f5f781556f0 2024-07-26T06:14:25.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.305+0000 7f5f6f400640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5f580762d0 0x7f5f58078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.305+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f5f70097b70 con 0x7f5f781556f0 2024-07-26T06:14:25.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.306+0000 7f5f76a00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5f580762d0 0x7f5f58078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.307+0000 7f5f76a00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5f580762d0 0x7f5f58078790 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5f68006fd0 tx=0x7f5f68008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:25.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.310+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f780af310 con 0x7f5f781556f0 2024-07-26T06:14:25.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.589+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f5f780b46f0 con 0x7f5f781556f0 2024-07-26T06:14:25.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.594+0000 7f5f6f400640 1 -- 172.21.15.163:0/2981569226 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f5f70014030 con 0x7f5f781556f0 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.596+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5f580762d0 msgr2=0x7f5f58078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.596+0000 7f5f77e00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5f580762d0 0x7f5f58078790 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5f68006fd0 tx=0x7f5f68008040 comp rx=0 tx=0).stop 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.596+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 msgr2=0x7f5f781e46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.596+0000 7f5f77e00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 0x7f5f781e46a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000cad0 tx=0x7f5f7000cf90 comp rx=0 tx=0).stop 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 shutdown_connections 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5f580762d0 0x7f5f58078790 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781556f0 0x7f5f781e46a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780ce4a0 0x7f5f781442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 --2- 172.21.15.163:0/2981569226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78071a70 0x7f5f78143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 >> 172.21.15.163:0/2981569226 conn(0x7f5f780b0e30 msgr2=0x7f5f780afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 shutdown_connections 2024-07-26T06:14:25.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.597+0000 7f5f77e00640 1 -- 172.21.15.163:0/2981569226 wait complete. 2024-07-26T06:14:25.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-07-26T06:14:25.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 -- 172.21.15.163:0/207804558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc06bf40 msgr2=0x7f29dc06c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 --2- 172.21.15.163:0/207804558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc06bf40 0x7f29dc06c320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f29dc070bc0 tx=0x7f29d402a8f0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 -- 172.21.15.163:0/207804558 shutdown_connections 2024-07-26T06:14:25.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 --2- 172.21.15.163:0/207804558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc06bf40 0x7f29dc06c320 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 --2- 172.21.15.163:0/207804558 >> v2:172.21.15.164:3300/0 conn(0x7f29dc0cb7e0 0x7f29dc15ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 -- 172.21.15.163:0/207804558 >> 172.21.15.163:0/207804558 conn(0x7f29dc06d420 msgr2=0x7f29dc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:25.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 -- 172.21.15.163:0/207804558 shutdown_connections 2024-07-26T06:14:25.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.824+0000 7f29e3600640 1 -- 172.21.15.163:0/207804558 wait complete. 2024-07-26T06:14:25.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.826+0000 7f29e3600640 1 Processor -- start 2024-07-26T06:14:25.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.826+0000 7f29e3600640 1 -- start start 2024-07-26T06:14:25.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29e3600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29dc06bf40 0x7f29dc150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29e3600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 0x7f29dc155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29e3600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29dc0cb7e0 0x7f29dc153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29e3600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc0736b0 con 0x7f29dc071cf0 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29e3600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc073530 con 0x7f29dc06bf40 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29e3600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f29dc073830 con 0x7f29dc0cb7e0 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29dbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 0x7f29dc155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29dbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 0x7f29dc155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49076/0 (socket says 172.21.15.163:49076) 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.827+0000 7f29dbe00640 1 -- 172.21.15.163:0/1052215078 learned_addr learned my addr 172.21.15.163:0/1052215078 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29e1600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29dc0cb7e0 0x7f29dc153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29e0c00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29dc06bf40 0x7f29dc150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29dbe00640 1 -- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29dc0cb7e0 msgr2=0x7f29dc153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29dbe00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29dc0cb7e0 0x7f29dc153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29dbe00640 1 -- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29dc06bf40 msgr2=0x7f29dc150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:25.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29dbe00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29dc06bf40 0x7f29dc150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:25.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29dbe00640 1 -- 172.21.15.163:0/1052215078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc153f10 con 0x7f29dc071cf0 2024-07-26T06:14:25.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29e0c00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29dc06bf40 0x7f29dc150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:25.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29dbe00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 0x7f29dc155840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c400 tx=0x7f29c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4017070 con 0x7f29dc071cf0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29c40035f0 con 0x7f29dc071cf0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29dc1541f0 con 0x7f29dc071cf0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.828+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4003a80 con 0x7f29dc071cf0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.829+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29dc158eb0 con 0x7f29dc071cf0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.830+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29dc0b22d0 con 0x7f29dc071cf0 2024-07-26T06:14:25.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.831+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f29c401a070 con 0x7f29dc071cf0 2024-07-26T06:14:25.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.832+0000 7f29d8c00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f29b80762d0 0x7f29b8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:25.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.832+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f29c4096670 con 0x7f29dc071cf0 2024-07-26T06:14:25.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.833+0000 7f29e0c00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f29b80762d0 0x7f29b8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:25.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.834+0000 7f29e0c00640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f29b80762d0 0x7f29b8078790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0058c0 tx=0x7f29cc005cb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:25.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:25.836+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29dc0b22d0 con 0x7f29dc071cf0 2024-07-26T06:14:25.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:14:25.881 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:14:26.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.119+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f29dc073d50 con 0x7f29dc071cf0 2024-07-26T06:14:26.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.123+0000 7f29d8c00640 1 -- 172.21.15.163:0/1052215078 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f29c40683a0 con 0x7f29dc071cf0 2024-07-26T06:14:26.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f29b80762d0 msgr2=0x7f29b8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f29b80762d0 0x7f29b8078790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0058c0 tx=0x7f29cc005cb0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 msgr2=0x7f29dc155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 0x7f29dc155840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c400 tx=0x7f29c400c8c0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 shutdown_connections 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f29b80762d0 0x7f29b8078790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29dc0cb7e0 0x7f29dc153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29dc071cf0 0x7f29dc155840 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 --2- 172.21.15.163:0/1052215078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29dc06bf40 0x7f29dc150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 >> 172.21.15.163:0/1052215078 conn(0x7f29dc06d420 msgr2=0x7f29dc0af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 shutdown_connections 2024-07-26T06:14:26.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.126+0000 7f29e3600640 1 -- 172.21.15.163:0/1052215078 wait complete. 2024-07-26T06:14:26.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.346+0000 7f73b6200640 1 -- 172.21.15.163:0/3780754491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00ce830 msgr2=0x7f73b00b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.346+0000 7f73b6200640 1 --2- 172.21.15.163:0/3780754491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00ce830 0x7f73b00b64c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f739c005fd0 tx=0x7f739c02a410 comp rx=0 tx=0).stop 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.347+0000 7f73b6200640 1 -- 172.21.15.163:0/3780754491 shutdown_connections 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.347+0000 7f73b6200640 1 --2- 172.21.15.163:0/3780754491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00ce830 0x7f73b00b64c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.347+0000 7f73b6200640 1 --2- 172.21.15.163:0/3780754491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015d5a0 0x7f73b00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.347+0000 7f73b6200640 1 -- 172.21.15.163:0/3780754491 >> 172.21.15.163:0/3780754491 conn(0x7f73b00cfda0 msgr2=0x7f73b00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:26.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.347+0000 7f73b6200640 1 -- 172.21.15.163:0/3780754491 shutdown_connections 2024-07-26T06:14:26.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.347+0000 7f73b6200640 1 -- 172.21.15.163:0/3780754491 wait complete. 2024-07-26T06:14:26.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.349+0000 7f73b6200640 1 Processor -- start 2024-07-26T06:14:26.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.349+0000 7f73b6200640 1 -- start start 2024-07-26T06:14:26.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.350+0000 7f73b6200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 0x7f73b01550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.350+0000 7f73b6200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b00ce830 0x7f73b014d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.350+0000 7f73b6200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b015d5a0 0x7f73b01478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.350+0000 7f73b6200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f73b00b5130 con 0x7f73b00b1e80 2024-07-26T06:14:26.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.350+0000 7f73b6200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f73b00b4fb0 con 0x7f73b015d5a0 2024-07-26T06:14:26.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.350+0000 7f73b6200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f73b00b52b0 con 0x7f73b00ce830 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 0x7f73b01550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 0x7f73b01550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49090/0 (socket says 172.21.15.163:49090) 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 -- 172.21.15.163:0/2100870421 learned_addr learned my addr 172.21.15.163:0/2100870421 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b5800640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b015d5a0 0x7f73b01478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73afe00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b00ce830 0x7f73b014d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 -- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b00ce830 msgr2=0x7f73b014d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b00ce830 0x7f73b014d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 -- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b015d5a0 msgr2=0x7f73b01478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b015d5a0 0x7f73b01478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b4e00640 1 -- 172.21.15.163:0/2100870421 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b00c7630 con 0x7f73b00b1e80 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.351+0000 7f73b5800640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b015d5a0 0x7f73b01478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.352+0000 7f73b4e00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 0x7f73b01550a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f73a400c9a0 tx=0x7f73a400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.352+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4007bb0 con 0x7f73b00b1e80 2024-07-26T06:14:26.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.352+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73a4005020 con 0x7f73b00b1e80 2024-07-26T06:14:26.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.352+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b00c7910 con 0x7f73b00b1e80 2024-07-26T06:14:26.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.352+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4007e60 con 0x7f73b00b1e80 2024-07-26T06:14:26.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.352+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b00c7ed0 con 0x7f73b00b1e80 2024-07-26T06:14:26.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.353+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b00cf5a0 con 0x7f73b00b1e80 2024-07-26T06:14:26.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.355+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f73a4005470 con 0x7f73b00b1e80 2024-07-26T06:14:26.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.356+0000 7f73acc00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f73900762d0 0x7f7390078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.356+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f73a40978f0 con 0x7f73b00b1e80 2024-07-26T06:14:26.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.356+0000 7f73afe00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f73900762d0 0x7f7390078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.358+0000 7f73afe00640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f73900762d0 0x7f7390078790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f739c033000 tx=0x7f739c004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:26.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.360+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73b00cf5a0 con 0x7f73b00b1e80 2024-07-26T06:14:26.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.646+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f73b00ce2f0 con 0x7f73b00b1e80 2024-07-26T06:14:26.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.651+0000 7f73acc00640 1 -- 172.21.15.163:0/2100870421 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v8) ==== 97+0+0 (secure 0 0 0) 0x7f73a4014030 con 0x7f73b00b1e80 2024-07-26T06:14:26.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f73900762d0 msgr2=0x7f7390078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f73900762d0 0x7f7390078790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f739c033000 tx=0x7f739c004730 comp rx=0 tx=0).stop 2024-07-26T06:14:26.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 msgr2=0x7f73b01550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 0x7f73b01550a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f73a400c9a0 tx=0x7f73a400ce60 comp rx=0 tx=0).stop 2024-07-26T06:14:26.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 shutdown_connections 2024-07-26T06:14:26.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f73900762d0 0x7f7390078790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b015d5a0 0x7f73b01478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b00ce830 0x7f73b014d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 --2- 172.21.15.163:0/2100870421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f73b00b1e80 0x7f73b01550a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.654+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 >> 172.21.15.163:0/2100870421 conn(0x7f73b00cfda0 msgr2=0x7f73b015b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:26.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.655+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 shutdown_connections 2024-07-26T06:14:26.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.655+0000 7f73b6200640 1 -- 172.21.15.163:0/2100870421 wait complete. 2024-07-26T06:14:26.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-07-26T06:14:26.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.872+0000 7f1bbee00640 1 -- 172.21.15.163:0/390338501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b9c003a30 msgr2=0x7f1b9c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.872+0000 7f1bbee00640 1 --2- 172.21.15.163:0/390338501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b9c003a30 0x7f1b9c003e30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c4a0 tx=0x7f1b9c005830 comp rx=0 tx=0).stop 2024-07-26T06:14:26.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.872+0000 7f1bbce00640 1 -- 172.21.15.163:0/390338501 reap_dead start 2024-07-26T06:14:26.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.872+0000 7f1bbee00640 1 -- 172.21.15.163:0/390338501 shutdown_connections 2024-07-26T06:14:26.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.872+0000 7f1bbee00640 1 -- 172.21.15.163:0/390338501 >> 172.21.15.163:0/390338501 conn(0x7f1bb80d36b0 msgr2=0x7f1bb80d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:26.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.873+0000 7f1bbee00640 1 -- 172.21.15.163:0/390338501 shutdown_connections 2024-07-26T06:14:26.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.873+0000 7f1bbee00640 1 -- 172.21.15.163:0/390338501 wait complete. 2024-07-26T06:14:26.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.874+0000 7f1bbee00640 1 Processor -- start 2024-07-26T06:14:26.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.874+0000 7f1bbee00640 1 -- start start 2024-07-26T06:14:26.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.875+0000 7f1bbee00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c003a30 0x7f1bb815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.875+0000 7f1bbee00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bb80aadd0 0x7f1bb80ac430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.875+0000 7f1bbee00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 0x7f1bb80ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.875+0000 7f1bbee00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80b6e30 con 0x7f1bb80ab230 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.875+0000 7f1bbee00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80ceca0 con 0x7f1bb80aadd0 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.875+0000 7f1bbee00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb80b6fb0 con 0x7f1b9c003a30 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 0x7f1bb80ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 0x7f1bb80ab870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49100/0 (socket says 172.21.15.163:49100) 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 -- 172.21.15.163:0/2699769374 learned_addr learned my addr 172.21.15.163:0/2699769374 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bb7e00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c003a30 0x7f1bb815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 -- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c003a30 msgr2=0x7f1bb815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c003a30 0x7f1bb815a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bb7400640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bb80aadd0 0x7f1bb80ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 -- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bb80aadd0 msgr2=0x7f1bb80ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:26.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bb80aadd0 0x7f1bb80ac430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:26.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 -- 172.21.15.163:0/2699769374 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb80aa050 con 0x7f1bb80ab230 2024-07-26T06:14:26.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bbce00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 0x7f1bb80ab870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8161000 tx=0x7f1bb000bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:26.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.876+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0019070 con 0x7f1bb80ab230 2024-07-26T06:14:26.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.877+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bb0004320 con 0x7f1bb80ab230 2024-07-26T06:14:26.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.877+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000f950 con 0x7f1bb80ab230 2024-07-26T06:14:26.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.877+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb80aa2f0 con 0x7f1bb80ab230 2024-07-26T06:14:26.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.877+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb80aa8b0 con 0x7f1bb80ab230 2024-07-26T06:14:26.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.878+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb80aaab0 con 0x7f1bb80ab230 2024-07-26T06:14:26.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.879+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1bb001c070 con 0x7f1bb80ab230 2024-07-26T06:14:26.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.880+0000 7f1bafe00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1b980762d0 0x7f1b98078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:26.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.880+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f1bb00969b0 con 0x7f1bb80ab230 2024-07-26T06:14:26.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.881+0000 7f1bb7e00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1b980762d0 0x7f1b98078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:26.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.882+0000 7f1bb7e00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1b980762d0 0x7f1b98078790 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c470 tx=0x7f1ba8001b00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:26.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:26.884+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1bb80aaab0 con 0x7f1bb80ab230 2024-07-26T06:14:27.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.153+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f1bb80b6720 con 0x7f1bb80ab230 2024-07-26T06:14:27.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.158+0000 7f1bafe00640 1 -- 172.21.15.163:0/2699769374 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f1bb00605c0 con 0x7f1bb80ab230 2024-07-26T06:14:27.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1b980762d0 msgr2=0x7f1b98078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1b980762d0 0x7f1b98078790 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c470 tx=0x7f1ba8001b00 comp rx=0 tx=0).stop 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 msgr2=0x7f1bb80ab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 0x7f1bb80ab870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8161000 tx=0x7f1bb000bad0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 shutdown_connections 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1b980762d0 0x7f1b98078790 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bb80ab230 0x7f1bb80ab870 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bb80aadd0 0x7f1bb80ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 --2- 172.21.15.163:0/2699769374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c003a30 0x7f1bb815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.161+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 >> 172.21.15.163:0/2699769374 conn(0x7f1bb80d36b0 msgr2=0x7f1bb80b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.162+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 shutdown_connections 2024-07-26T06:14:27.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.162+0000 7f1bbee00640 1 -- 172.21.15.163:0/2699769374 wait complete. 2024-07-26T06:14:27.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:14:27.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 -- 172.21.15.163:0/889881335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea9015a6b0 msgr2=0x7fea900cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 --2- 172.21.15.163:0/889881335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea9015a6b0 0x7fea900cbab0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fea9006f6e0 tx=0x7fea7802a560 comp rx=0 tx=0).stop 2024-07-26T06:14:27.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 -- 172.21.15.163:0/889881335 shutdown_connections 2024-07-26T06:14:27.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 --2- 172.21.15.163:0/889881335 >> v2:172.21.15.164:3300/0 conn(0x7fea900cbff0 0x7fea9015d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 --2- 172.21.15.163:0/889881335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea9015a6b0 0x7fea900cbab0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 --2- 172.21.15.163:0/889881335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea900741f0 0x7fea900cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.377+0000 7fea94c00640 1 -- 172.21.15.163:0/889881335 >> 172.21.15.163:0/889881335 conn(0x7fea900b0e30 msgr2=0x7fea900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:27.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.378+0000 7fea94c00640 1 -- 172.21.15.163:0/889881335 shutdown_connections 2024-07-26T06:14:27.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.378+0000 7fea94c00640 1 -- 172.21.15.163:0/889881335 wait complete. 2024-07-26T06:14:27.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.379+0000 7fea94c00640 1 Processor -- start 2024-07-26T06:14:27.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.379+0000 7fea94c00640 1 -- start start 2024-07-26T06:14:27.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea94c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 0x7fea9014f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea94c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea900cbff0 0x7fea90150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea94c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea9015a6b0 0x7fea90150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea94c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea900745d0 con 0x7fea900741f0 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea94c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fea90074750 con 0x7fea9015a6b0 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea94c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fea900b2230 con 0x7fea900cbff0 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.380+0000 7fea8f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 0x7fea9014f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 0x7fea9014f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49112/0 (socket says 172.21.15.163:49112) 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 -- 172.21.15.163:0/626996679 learned_addr learned my addr 172.21.15.163:0/626996679 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea86a00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea900cbff0 0x7fea90150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8fe00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea9015a6b0 0x7fea90150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 -- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea900cbff0 msgr2=0x7fea90150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea900cbff0 0x7fea90150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 -- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea9015a6b0 msgr2=0x7fea90150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea9015a6b0 0x7fea90150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 -- 172.21.15.163:0/626996679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea90150e60 con 0x7fea900741f0 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8fe00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea9015a6b0 0x7fea90150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea8f400640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 0x7fea9014f670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fea8000cd10 tx=0x7fea80007610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.381+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80004b20 con 0x7fea900741f0 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.382+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea80004db0 con 0x7fea900741f0 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.382+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80005280 con 0x7fea900741f0 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.382+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea90154030 con 0x7fea900741f0 2024-07-26T06:14:27.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.382+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea901525f0 con 0x7fea900741f0 2024-07-26T06:14:27.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.383+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea900b27f0 con 0x7fea900741f0 2024-07-26T06:14:27.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.384+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fea80005420 con 0x7fea900741f0 2024-07-26T06:14:27.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.385+0000 7fea87e00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea6c0762d0 0x7fea6c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.385+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7fea80097b90 con 0x7fea900741f0 2024-07-26T06:14:27.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.385+0000 7fea86a00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea6c0762d0 0x7fea6c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.386+0000 7fea86a00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea6c0762d0 0x7fea6c078790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea78033000 tx=0x7fea78004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:27.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.389+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fea80014030 con 0x7fea900741f0 2024-07-26T06:14:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.597+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fea900b4aa0 con 0x7fea900741f0 2024-07-26T06:14:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.598+0000 7fea87e00640 1 -- 172.21.15.163:0/626996679 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) ==== 84+0+65 (secure 0 0 0) 0x7fea80061840 con 0x7fea900741f0 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:27.599 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea6c0762d0 msgr2=0x7fea6c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea6c0762d0 0x7fea6c078790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea78033000 tx=0x7fea78004730 comp rx=0 tx=0).stop 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 msgr2=0x7fea9014f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 0x7fea9014f670 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fea8000cd10 tx=0x7fea80007610 comp rx=0 tx=0).stop 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 shutdown_connections 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea6c0762d0 0x7fea6c078790 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea9015a6b0 0x7fea90150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea900cbff0 0x7fea90150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.600+0000 7fea94c00640 1 --2- 172.21.15.163:0/626996679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea900741f0 0x7fea9014f670 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.601+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 >> 172.21.15.163:0/626996679 conn(0x7fea900b0e30 msgr2=0x7fea900734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:27.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.601+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 shutdown_connections 2024-07-26T06:14:27.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.601+0000 7fea94c00640 1 -- 172.21.15.163:0/626996679 wait complete. 2024-07-26T06:14:27.619 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:14:27.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:14:27.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 -- 172.21.15.163:0/706237472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c80b1e80 msgr2=0x7f68c80b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 --2- 172.21.15.163:0/706237472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c80b1e80 0x7f68c80b6bd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f68c0005fd0 tx=0x7f68c002a4b0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 -- 172.21.15.163:0/706237472 shutdown_connections 2024-07-26T06:14:27.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 --2- 172.21.15.163:0/706237472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c80b1e80 0x7f68c80b6bd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 --2- 172.21.15.163:0/706237472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68c815d5a0 0x7f68c80ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 -- 172.21.15.163:0/706237472 >> 172.21.15.163:0/706237472 conn(0x7f68c80cfda0 msgr2=0x7f68c80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:27.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 -- 172.21.15.163:0/706237472 shutdown_connections 2024-07-26T06:14:27.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.828+0000 7f68cea00640 1 -- 172.21.15.163:0/706237472 wait complete. 2024-07-26T06:14:27.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.830+0000 7f68cea00640 1 Processor -- start 2024-07-26T06:14:27.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.830+0000 7f68cea00640 1 -- start start 2024-07-26T06:14:27.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68c80b1e80 0x7f68c80c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cea00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68c80ce830 0x7f68c81f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 0x7f68c815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cea00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f68c80b5130 con 0x7f68c815d5a0 2024-07-26T06:14:27.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cea00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f68c80b4fb0 con 0x7f68c80b1e80 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cea00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f68c80b52b0 con 0x7f68c80ce830 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 0x7f68c815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 0x7f68c815ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49122/0 (socket says 172.21.15.163:49122) 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 -- 172.21.15.163:0/4291297777 learned_addr learned my addr 172.21.15.163:0/4291297777 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ccc00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68c80ce830 0x7f68c81f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68cd600640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68c80b1e80 0x7f68c80c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 -- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68c80ce830 msgr2=0x7f68c81f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68c80ce830 0x7f68c81f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 -- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68c80b1e80 msgr2=0x7f68c80c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68c80b1e80 0x7f68c80c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.831+0000 7f68ce000640 1 -- 172.21.15.163:0/4291297777 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c81f38f0 con 0x7f68c815d5a0 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.832+0000 7f68ce000640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 0x7f68c815ff20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f68c0006010 tx=0x7f68c0003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.832+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c003f070 con 0x7f68c815d5a0 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.832+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68c00047a0 con 0x7f68c815d5a0 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.832+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0035030 con 0x7f68c815d5a0 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.832+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c81f7690 con 0x7f68c815d5a0 2024-07-26T06:14:27.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.832+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c81f7c50 con 0x7f68c815d5a0 2024-07-26T06:14:27.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.833+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c80ce2f0 con 0x7f68c815d5a0 2024-07-26T06:14:27.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.835+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f68c0042070 con 0x7f68c815d5a0 2024-07-26T06:14:27.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.836+0000 7f68c5600640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f68ac0762d0 0x7f68ac078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:27.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.836+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4857+0+0 (secure 0 0 0) 0x7f68c00b8330 con 0x7f68c815d5a0 2024-07-26T06:14:27.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.837+0000 7f68cd600640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f68ac0762d0 0x7f68ac078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:27.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.838+0000 7f68cd600640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f68ac0762d0 0x7f68ac078790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f68b8005fd0 tx=0x7f68b8005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:27.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:27.841+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f68c0081f40 con 0x7f68c815d5a0 2024-07-26T06:14:28.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.042+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f68c80b8990 con 0x7f68c815d5a0 2024-07-26T06:14:28.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.854+0000 7f68c5600640 1 -- 172.21.15.163:0/4291297777 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) ==== 138+0+0 (secure 0 0 0) 0x7f68c0081840 con 0x7f68c815d5a0 2024-07-26T06:14:28.856 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:14:28.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.858+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f68ac0762d0 msgr2=0x7f68ac078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:28.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.858+0000 7f68cea00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f68ac0762d0 0x7f68ac078790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f68b8005fd0 tx=0x7f68b8005850 comp rx=0 tx=0).stop 2024-07-26T06:14:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.858+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 msgr2=0x7f68c815ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.858+0000 7f68cea00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 0x7f68c815ff20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f68c0006010 tx=0x7f68c0003890 comp rx=0 tx=0).stop 2024-07-26T06:14:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.859+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 shutdown_connections 2024-07-26T06:14:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.859+0000 7f68cea00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f68ac0762d0 0x7f68ac078790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.859+0000 7f68cea00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f68c815d5a0 0x7f68c815ff20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.859+0000 7f68cea00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68c80ce830 0x7f68c81f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.859+0000 7f68cea00640 1 --2- 172.21.15.163:0/4291297777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68c80b1e80 0x7f68c80c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.859+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 >> 172.21.15.163:0/4291297777 conn(0x7f68c80cfda0 msgr2=0x7f68c815aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.861+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 shutdown_connections 2024-07-26T06:14:28.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:28.863+0000 7f68cea00640 1 -- 172.21.15.163:0/4291297777 wait complete. 2024-07-26T06:14:28.889 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e5600640 1 -- 172.21.15.163:0/2030524562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c4004d40 msgr2=0x7f14c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e5600640 1 --2- 172.21.15.163:0/2030524562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14c4004d40 0x7f14c40051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f14d8005fd0 tx=0x7f14c4005830 comp rx=0 tx=0).stop 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e4c00640 1 -- 172.21.15.163:0/2030524562 reap_dead start 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e5600640 1 -- 172.21.15.163:0/2030524562 shutdown_connections 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e5600640 1 -- 172.21.15.163:0/2030524562 >> 172.21.15.163:0/2030524562 conn(0x7f14e00b0e30 msgr2=0x7f14e00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e5600640 1 -- 172.21.15.163:0/2030524562 shutdown_connections 2024-07-26T06:14:29.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.096+0000 7f14e5600640 1 -- 172.21.15.163:0/2030524562 wait complete. 2024-07-26T06:14:29.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.097+0000 7f14e5600640 1 Processor -- start 2024-07-26T06:14:29.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 -- start start 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e0153370 0x7f14e014a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 0x7f14e014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e0153750 0x7f14e0153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14e00716b0 con 0x7f14e014b790 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f14e0071530 con 0x7f14e0153370 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.098+0000 7f14e5600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f14e0071830 con 0x7f14e0153750 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 0x7f14e014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:29.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 0x7f14e014faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49140/0 (socket says 172.21.15.163:49140) 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 -- 172.21.15.163:0/727751072 learned_addr learned my addr 172.21.15.163:0/727751072 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14dfe00640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e0153370 0x7f14e014a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 -- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e0153750 msgr2=0x7f14e0153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e0153750 0x7f14e0153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 -- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e0153370 msgr2=0x7f14e014a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e0153370 0x7f14e014a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 -- 172.21.15.163:0/727751072 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e014a6f0 con 0x7f14e014b790 2024-07-26T06:14:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.099+0000 7f14df400640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 0x7f14e014faa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00e740 tx=0x7f14cc00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:29.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.100+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00cbe0 con 0x7f14e014b790 2024-07-26T06:14:29.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.100+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14cc00ce70 con 0x7f14e014b790 2024-07-26T06:14:29.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.100+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e014a9d0 con 0x7f14e014b790 2024-07-26T06:14:29.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.100+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc011690 con 0x7f14e014b790 2024-07-26T06:14:29.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.100+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e014af90 con 0x7f14e014b790 2024-07-26T06:14:29.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.101+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e00ced60 con 0x7f14e014b790 2024-07-26T06:14:29.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.102+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f14cc011830 con 0x7f14e014b790 2024-07-26T06:14:29.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.103+0000 7f14d7e00640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14c00762d0 0x7f14c0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:29.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.103+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5227+0+0 (secure 0 0 0) 0x7f14cc097f40 con 0x7f14e014b790 2024-07-26T06:14:29.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.104+0000 7f14dfe00640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14c00762d0 0x7f14c0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:29.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.105+0000 7f14dfe00640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14c00762d0 0x7f14c0078790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14d0001860 tx=0x7f14d0001c00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:29.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.108+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14cc016030 con 0x7f14e014b790 2024-07-26T06:14:29.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.318+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f14e0154110 con 0x7f14e014b790 2024-07-26T06:14:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.861+0000 7f14d7e00640 1 -- 172.21.15.163:0/727751072 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) ==== 170+0+0 (secure 0 0 0) 0x7f14cc061b10 con 0x7f14e014b790 2024-07-26T06:14:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:14:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14c00762d0 msgr2=0x7f14c0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14c00762d0 0x7f14c0078790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14d0001860 tx=0x7f14d0001c00 comp rx=0 tx=0).stop 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 msgr2=0x7f14e014faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 0x7f14e014faa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00e740 tx=0x7f14cc00ec00 comp rx=0 tx=0).stop 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 shutdown_connections 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14c00762d0 0x7f14c0078790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e0153750 0x7f14e0153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14e014b790 0x7f14e014faa0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 --2- 172.21.15.163:0/727751072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e0153370 0x7f14e014a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.866+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 >> 172.21.15.163:0/727751072 conn(0x7f14e00b0e30 msgr2=0x7f14e00cdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:29.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.868+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 shutdown_connections 2024-07-26T06:14:29.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:29.871+0000 7f14e5600640 1 -- 172.21.15.163:0/727751072 wait complete. 2024-07-26T06:14:29.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 -- 172.21.15.163:0/4177165366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 msgr2=0x7f62080cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 --2- 172.21.15.163:0/4177165366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f62080cc140 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f61f8005fd0 tx=0x7f61f802a530 comp rx=0 tx=0).stop 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 -- 172.21.15.163:0/4177165366 shutdown_connections 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 --2- 172.21.15.163:0/4177165366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f620815b240 0x7f620815b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 --2- 172.21.15.163:0/4177165366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f62080cc140 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 -- 172.21.15.163:0/4177165366 >> 172.21.15.163:0/4177165366 conn(0x7f620806d420 msgr2=0x7f620806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 -- 172.21.15.163:0/4177165366 shutdown_connections 2024-07-26T06:14:30.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.085+0000 7f6210800640 1 -- 172.21.15.163:0/4177165366 wait complete. 2024-07-26T06:14:30.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.086+0000 7f6210800640 1 Processor -- start 2024-07-26T06:14:30.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 -- start start 2024-07-26T06:14:30.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f6208149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:30.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62081507d0 0x7f62080af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:30.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f620815b240 0x7f6208154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:30.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f620806c0c0 con 0x7f62080cbcb0 2024-07-26T06:14:30.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f620806bf40 con 0x7f620815b240 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.087+0000 7f6210800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f620806c240 con 0x7f62081507d0 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f6208149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f6208149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49162/0 (socket says 172.21.15.163:49162) 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 -- 172.21.15.163:0/3281874868 learned_addr learned my addr 172.21.15.163:0/3281874868 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620e800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f620815b240 0x7f6208154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 -- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62081507d0 msgr2=0x7f62080af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62081507d0 0x7f62080af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 -- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f620815b240 msgr2=0x7f6208154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f620815b240 0x7f6208154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.088+0000 7f620de00640 1 -- 172.21.15.163:0/3281874868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620814de80 con 0x7f62080cbcb0 2024-07-26T06:14:30.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.089+0000 7f620de00640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f6208149fb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c410 tx=0x7f61fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:30.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.089+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc017070 con 0x7f62080cbcb0 2024-07-26T06:14:30.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.089+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61fc0035f0 con 0x7f62080cbcb0 2024-07-26T06:14:30.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.089+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc003a80 con 0x7f62080cbcb0 2024-07-26T06:14:30.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.089+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620814d8c0 con 0x7f62080cbcb0 2024-07-26T06:14:30.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.089+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6208154590 con 0x7f62080cbcb0 2024-07-26T06:14:30.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.090+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62080cc140 con 0x7f62080cbcb0 2024-07-26T06:14:30.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.092+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f61fc01a070 con 0x7f62080cbcb0 2024-07-26T06:14:30.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.092+0000 7f6202000640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f61ec0762d0 0x7f61ec078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:30.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.093+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5609+0+0 (secure 0 0 0) 0x7f61fc096fb0 con 0x7f62080cbcb0 2024-07-26T06:14:30.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.093+0000 7f620d400640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f61ec0762d0 0x7f61ec078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:30.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.095+0000 7f620d400640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f61ec0762d0 0x7f61ec078790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f61f8033000 tx=0x7f61f80046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:30.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.096+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f61fc021020 con 0x7f62080cbcb0 2024-07-26T06:14:30.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.335+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f620814f2d0 con 0x7f62080cbcb0 2024-07-26T06:14:30.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.890+0000 7f6202000640 1 -- 172.21.15.163:0/3281874868 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7f61fc060950 con 0x7f62080cbcb0 2024-07-26T06:14:30.891 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 4 and data pool 5 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f61ec0762d0 msgr2=0x7f61ec078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f61ec0762d0 0x7f61ec078790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f61f8033000 tx=0x7f61f80046c0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 msgr2=0x7f6208149fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f6208149fb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c410 tx=0x7f61fc00c8d0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 shutdown_connections 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f61ec0762d0 0x7f61ec078790 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f620815b240 0x7f6208154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62081507d0 0x7f62080af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 --2- 172.21.15.163:0/3281874868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f62080cbcb0 0x7f6208149fb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:30.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 >> 172.21.15.163:0/3281874868 conn(0x7f620806d420 msgr2=0x7f62080b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:30.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 shutdown_connections 2024-07-26T06:14:30.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:30.898+0000 7f6210800640 1 -- 172.21.15.163:0/3281874868 wait complete. 2024-07-26T06:14:30.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 -- 172.21.15.163:0/2515107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f40cb7e0 msgr2=0x7f25f415ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 --2- 172.21.15.163:0/2515107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f40cb7e0 0x7f25f415ed20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f25e4005fd0 tx=0x7f25e402a820 comp rx=0 tx=0).stop 2024-07-26T06:14:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 -- 172.21.15.163:0/2515107220 shutdown_connections 2024-07-26T06:14:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 --2- 172.21.15.163:0/2515107220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f40cb7e0 0x7f25f415ed20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 -- 172.21.15.163:0/2515107220 >> 172.21.15.163:0/2515107220 conn(0x7f25f406d420 msgr2=0x7f25f406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:31.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 -- 172.21.15.163:0/2515107220 shutdown_connections 2024-07-26T06:14:31.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.113+0000 7f25fb800640 1 -- 172.21.15.163:0/2515107220 wait complete. 2024-07-26T06:14:31.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.115+0000 7f25fb800640 1 Processor -- start 2024-07-26T06:14:31.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.115+0000 7f25fb800640 1 -- start start 2024-07-26T06:14:31.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25fb800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 0x7f25f4150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25fb800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25f4071cf0 0x7f25f414f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25fb800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25f40cb7e0 0x7f25f414ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25fb800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f25f40738c0 con 0x7f25f406bf40 2024-07-26T06:14:31.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25fb800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f25f4073740 con 0x7f25f40cb7e0 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25fb800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f25f4073a40 con 0x7f25f4071cf0 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 0x7f25f4150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 0x7f25f4150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49180/0 (socket says 172.21.15.163:49180) 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 -- 172.21.15.163:0/120985689 learned_addr learned my addr 172.21.15.163:0/120985689 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 -- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25f4071cf0 msgr2=0x7f25f414f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25f4071cf0 0x7f25f414f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 -- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25f40cb7e0 msgr2=0x7f25f414ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25f40cb7e0 0x7f25f414ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.116+0000 7f25f8e00640 1 -- 172.21.15.163:0/120985689 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f41517c0 con 0x7f25f406bf40 2024-07-26T06:14:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.117+0000 7f25f8e00640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 0x7f25f4150d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f25e4035040 tx=0x7f25e4032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:31.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.117+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e4037040 con 0x7f25f406bf40 2024-07-26T06:14:31.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.117+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25e4042070 con 0x7f25f406bf40 2024-07-26T06:14:31.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.117+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f4151aa0 con 0x7f25f406bf40 2024-07-26T06:14:31.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.117+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e40042a0 con 0x7f25f406bf40 2024-07-26T06:14:31.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.117+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f4152060 con 0x7f25f406bf40 2024-07-26T06:14:31.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.118+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f41523a0 con 0x7f25f406bf40 2024-07-26T06:14:31.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.119+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25e4004440 con 0x7f25f406bf40 2024-07-26T06:14:31.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.120+0000 7f25f0c00640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25d40762d0 0x7f25d4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.120+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f25e40b84c0 con 0x7f25f406bf40 2024-07-26T06:14:31.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.122+0000 7f25f3e00640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25d40762d0 0x7f25d4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.123+0000 7f25f3e00640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25d40762d0 0x7f25d4078790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25f414dd00 tx=0x7f25e0007990 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:31.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.125+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f25e4081eb0 con 0x7f25f406bf40 2024-07-26T06:14:31.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.319+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25f415ed20 con 0x7f25f406bf40 2024-07-26T06:14:31.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.322+0000 7f25f0c00640 1 -- 172.21.15.163:0/120985689 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16706 (secure 0 0 0) 0x7f25e40817b0 con 0x7f25f406bf40 2024-07-26T06:14:31.324 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:31.324 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":33,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:30.337649+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:14:28.059274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-07-26T06:14:29.335706+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.324+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25d40762d0 msgr2=0x7f25d4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.324+0000 7f25fb800640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25d40762d0 0x7f25d4078790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25f414dd00 tx=0x7f25e0007990 comp rx=0 tx=0).stop 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.324+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 msgr2=0x7f25f4150d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.324+0000 7f25fb800640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 0x7f25f4150d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f25e4035040 tx=0x7f25e4032c90 comp rx=0 tx=0).stop 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 shutdown_connections 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25d40762d0 0x7f25d4078790 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25f40cb7e0 0x7f25f414ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25f4071cf0 0x7f25f414f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 --2- 172.21.15.163:0/120985689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25f406bf40 0x7f25f4150d70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 >> 172.21.15.163:0/120985689 conn(0x7f25f406d420 msgr2=0x7f25f40b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:31.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 shutdown_connections 2024-07-26T06:14:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.325+0000 7f25fb800640 1 -- 172.21.15.163:0/120985689 wait complete. 2024-07-26T06:14:31.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 -- 172.21.15.163:0/127735745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89141556f0 msgr2=0x7f89140cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 --2- 172.21.15.163:0/127735745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89141556f0 0x7f89140cdd90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8904005fd0 tx=0x7f890402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 -- 172.21.15.163:0/127735745 shutdown_connections 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 --2- 172.21.15.163:0/127735745 >> v2:172.21.15.164:3300/0 conn(0x7f89140ce2d0 0x7f8914071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 --2- 172.21.15.163:0/127735745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89141556f0 0x7f89140cdd90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 -- 172.21.15.163:0/127735745 >> 172.21.15.163:0/127735745 conn(0x7f89140b0e30 msgr2=0x7f89140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 -- 172.21.15.163:0/127735745 shutdown_connections 2024-07-26T06:14:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.541+0000 7f8919c00640 1 -- 172.21.15.163:0/127735745 wait complete. 2024-07-26T06:14:31.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.543+0000 7f8919c00640 1 Processor -- start 2024-07-26T06:14:31.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.543+0000 7f8919c00640 1 -- start start 2024-07-26T06:14:31.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8914071a70 0x7f89141f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89140ce2d0 0x7f89141e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 0x7f89141e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f891415f310 con 0x7f89141556f0 2024-07-26T06:14:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8914071ee0 con 0x7f89140ce2d0 2024-07-26T06:14:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f891415f490 con 0x7f8914071a70 2024-07-26T06:14:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.544+0000 7f8919200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 0x7f89141e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 0x7f89141e58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49190/0 (socket says 172.21.15.163:49190) 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 -- 172.21.15.163:0/478707769 learned_addr learned my addr 172.21.15.163:0/478707769 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8913e00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8914071a70 0x7f89141f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f890be00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89140ce2d0 0x7f89141e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 -- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8914071a70 msgr2=0x7f89141f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8914071a70 0x7f89141f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 -- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89140ce2d0 msgr2=0x7f89141e92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89140ce2d0 0x7f89141e92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 -- 172.21.15.163:0/478707769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89141f55f0 con 0x7f89141556f0 2024-07-26T06:14:31.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f890be00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89140ce2d0 0x7f89141e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8919200640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 0x7f89141e58b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f890c00c7a0 tx=0x7f890c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.545+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c003b70 con 0x7f89141556f0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.546+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f890c003d30 con 0x7f89141556f0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.546+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c00aab0 con 0x7f89141556f0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.546+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89141e4810 con 0x7f89141556f0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.546+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89141e4dd0 con 0x7f89141556f0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.547+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8914071530 con 0x7f89141556f0 2024-07-26T06:14:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.548+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f890c00ac50 con 0x7f89141556f0 2024-07-26T06:14:31.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.549+0000 7f8910c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88f00762d0 0x7f88f0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.549+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f890c097640 con 0x7f89141556f0 2024-07-26T06:14:31.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.549+0000 7f8913e00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88f00762d0 0x7f88f0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.550+0000 7f8913e00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88f00762d0 0x7f88f0078790 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8904033000 tx=0x7f8904004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:31.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.553+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f89141f55f0 con 0x7f89141556f0 2024-07-26T06:14:31.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.743+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89140b34f0 con 0x7f89141556f0 2024-07-26T06:14:31.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.746+0000 7f8910c00640 1 -- 172.21.15.163:0/478707769 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16706 (secure 0 0 0) 0x7f890c060f90 con 0x7f89141556f0 2024-07-26T06:14:31.747 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:31.747 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":33,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:30.337649+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:14:28.059274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-07-26T06:14:29.335706+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.748+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88f00762d0 msgr2=0x7f88f0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.748+0000 7f8919c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88f00762d0 0x7f88f0078790 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8904033000 tx=0x7f8904004730 comp rx=0 tx=0).stop 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.748+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 msgr2=0x7f89141e58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.748+0000 7f8919c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 0x7f89141e58b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f890c00c7a0 tx=0x7f890c00cc60 comp rx=0 tx=0).stop 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 shutdown_connections 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88f00762d0 0x7f88f0078790 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89141556f0 0x7f89141e58b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89140ce2d0 0x7f89141e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 --2- 172.21.15.163:0/478707769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8914071a70 0x7f89141f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 >> 172.21.15.163:0/478707769 conn(0x7f89140b0e30 msgr2=0x7f89140b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:31.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 shutdown_connections 2024-07-26T06:14:31.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.749+0000 7f8919c00640 1 -- 172.21.15.163:0/478707769 wait complete. 2024-07-26T06:14:31.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:14:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.982+0000 7f35d2000640 1 -- 172.21.15.163:0/2647298990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 msgr2=0x7f35cc0b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.982+0000 7f35d2000640 1 --2- 172.21.15.163:0/2647298990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc0b5270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f35cc06f200 tx=0x7f35b402a4d0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 -- 172.21.15.163:0/2647298990 shutdown_connections 2024-07-26T06:14:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 --2- 172.21.15.163:0/2647298990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc0c9ec0 0x7f35cc0b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 --2- 172.21.15.163:0/2647298990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc0b5270 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 --2- 172.21.15.163:0/2647298990 >> v2:172.21.15.164:3300/0 conn(0x7f35cc15d310 0x7f35cc0b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 -- 172.21.15.163:0/2647298990 >> 172.21.15.163:0/2647298990 conn(0x7f35cc0cfd30 msgr2=0x7f35cc0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 -- 172.21.15.163:0/2647298990 shutdown_connections 2024-07-26T06:14:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.983+0000 7f35d2000640 1 -- 172.21.15.163:0/2647298990 wait complete. 2024-07-26T06:14:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.984+0000 7f35d2000640 1 Processor -- start 2024-07-26T06:14:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 -- start start 2024-07-26T06:14:31.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35cc0c9ec0 0x7f35cc0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc15d310 0x7f35cc0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0d1000 con 0x7f35cc0ce960 2024-07-26T06:14:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0d0e80 con 0x7f35cc0c9ec0 2024-07-26T06:14:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.985+0000 7f35d2000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc0d1180 con 0x7f35cc15d310 2024-07-26T06:14:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49208/0 (socket says 172.21.15.163:49208) 2024-07-26T06:14:31.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 -- 172.21.15.163:0/3810334799 learned_addr learned my addr 172.21.15.163:0/3810334799 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:31.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35d0c00640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35cc0c9ec0 0x7f35cc0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35d1600640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc15d310 0x7f35cc0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 -- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc15d310 msgr2=0x7f35cc0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc15d310 0x7f35cc0ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 -- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35cc0c9ec0 msgr2=0x7f35cc0cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35cc0c9ec0 0x7f35cc0cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35c3e00640 1 -- 172.21.15.163:0/3810334799 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc0a9d80 con 0x7f35cc0ce960 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.986+0000 7f35d1600640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc15d310 0x7f35cc0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:31.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.987+0000 7f35c3e00640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc159b20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009850 tx=0x7f35b40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.987+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b403f070 con 0x7f35cc0ce960 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.987+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35b4004890 con 0x7f35cc0ce960 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.987+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc0aa060 con 0x7f35cc0ce960 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.987+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4035030 con 0x7f35cc0ce960 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.987+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc0aa620 con 0x7f35cc0ce960 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.988+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35cc0d1710 con 0x7f35cc0ce960 2024-07-26T06:14:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.989+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f35b4042070 con 0x7f35cc0ce960 2024-07-26T06:14:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.990+0000 7f35c9600640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f35a80762d0 0x7f35a8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.990+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f35b40b8780 con 0x7f35cc0ce960 2024-07-26T06:14:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.991+0000 7f35d0c00640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f35a80762d0 0x7f35a8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.992+0000 7f35d0c00640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f35a80762d0 0x7f35a8078790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00b810 tx=0x7f35bc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:31.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:31.994+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f35b4082050 con 0x7f35cc0ce960 2024-07-26T06:14:32.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.245+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f35cc0b22f0 con 0x7f35cc0ce960 2024-07-26T06:14:32.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.927+0000 7f35c9600640 1 -- 172.21.15.163:0/3810334799 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f35b4081950 con 0x7f35cc0ce960 2024-07-26T06:14:32.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f35a80762d0 msgr2=0x7f35a8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f35a80762d0 0x7f35a8078790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00b810 tx=0x7f35bc008040 comp rx=0 tx=0).stop 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 msgr2=0x7f35cc159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc159b20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009850 tx=0x7f35b40039e0 comp rx=0 tx=0).stop 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 shutdown_connections 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f35a80762d0 0x7f35a8078790 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35cc15d310 0x7f35cc0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35cc0ce960 0x7f35cc159b20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 --2- 172.21.15.163:0/3810334799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35cc0c9ec0 0x7f35cc0cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.931+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 >> 172.21.15.163:0/3810334799 conn(0x7f35cc0cfd30 msgr2=0x7f35cc0b7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.932+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 shutdown_connections 2024-07-26T06:14:32.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:32.932+0000 7f35d2000640 1 -- 172.21.15.163:0/3810334799 wait complete. 2024-07-26T06:14:32.957 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:14:32.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:14:33.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.166+0000 7f72af400640 1 -- 172.21.15.163:0/766126096 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72980047a0 con 0x7f72b00ae870 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 -- 172.21.15.163:0/766126096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00ae870 msgr2=0x7f72b00b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 --2- 172.21.15.163:0/766126096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00ae870 0x7f72b00b3fb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7298005fd0 tx=0x7f729802a540 comp rx=0 tx=0).stop 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 -- 172.21.15.163:0/766126096 shutdown_connections 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 --2- 172.21.15.163:0/766126096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00ae870 0x7f72b00b3fb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 --2- 172.21.15.163:0/766126096 >> v2:172.21.15.164:3300/0 conn(0x7f72b00b3bd0 0x7f72b00b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 -- 172.21.15.163:0/766126096 >> 172.21.15.163:0/766126096 conn(0x7f72b00b0e30 msgr2=0x7f72b00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 -- 172.21.15.163:0/766126096 shutdown_connections 2024-07-26T06:14:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.167+0000 7f72b5600640 1 -- 172.21.15.163:0/766126096 wait complete. 2024-07-26T06:14:33.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.169+0000 7f72b5600640 1 Processor -- start 2024-07-26T06:14:33.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.169+0000 7f72b5600640 1 -- start start 2024-07-26T06:14:33.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b5600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 0x7f72b0151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:33.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b5600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00b3bd0 0x7f72b01498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:33.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b5600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72b015db10 0x7f72b0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:33.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b5600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72b00cb870 con 0x7f72b00ae870 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b5600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f72b0073300 con 0x7f72b015db10 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b5600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f72b00cb9f0 con 0x7f72b00b3bd0 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 0x7f72b0151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 0x7f72b0151840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49220/0 (socket says 172.21.15.163:49220) 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72afe00640 1 -- 172.21.15.163:0/2612519139 learned_addr learned my addr 172.21.15.163:0/2612519139 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72a7400640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00b3bd0 0x7f72b01498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.170+0000 7f72b4c00640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72b015db10 0x7f72b0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72afe00640 1 -- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00b3bd0 msgr2=0x7f72b01498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72afe00640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00b3bd0 0x7f72b01498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72afe00640 1 -- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72b015db10 msgr2=0x7f72b0152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72afe00640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72b015db10 0x7f72b0152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72afe00640 1 -- 172.21.15.163:0/2612519139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b0153080 con 0x7f72b00ae870 2024-07-26T06:14:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72afe00640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 0x7f72b0151840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f72a000d950 tx=0x7f72a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0017070 con 0x7f72b00ae870 2024-07-26T06:14:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72a0009d20 con 0x7f72b00ae870 2024-07-26T06:14:33.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00106c0 con 0x7f72b00ae870 2024-07-26T06:14:33.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b0150d60 con 0x7f72b00ae870 2024-07-26T06:14:33.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.171+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b0151d80 con 0x7f72b00ae870 2024-07-26T06:14:33.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.172+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b01575c0 con 0x7f72b00ae870 2024-07-26T06:14:33.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.174+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f72a0010860 con 0x7f72b00ae870 2024-07-26T06:14:33.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.175+0000 7f72acc00640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f728c0762d0 0x7f728c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:33.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.175+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f72a0097b80 con 0x7f72b00ae870 2024-07-26T06:14:33.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.175+0000 7f72a7400640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f728c0762d0 0x7f728c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:33.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.176+0000 7f72a7400640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f728c0762d0 0x7f728c078790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7298033000 tx=0x7f7298004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:33.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.179+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f72b01575c0 con 0x7f72b00ae870 2024-07-26T06:14:33.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.426+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f72b00b26d0 con 0x7f72b00ae870 2024-07-26T06:14:33.822 INFO:tasks.ceph.mds.a.smithi163.stderr:2024-07-26T06:14:33.822+0000 7fcdb0c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:14:33.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.941+0000 7f72acc00640 1 -- 172.21.15.163:0/2612519139 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f72a0061440 con 0x7f72b00ae870 2024-07-26T06:14:33.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.944+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f728c0762d0 msgr2=0x7f728c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:33.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.944+0000 7f72b5600640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f728c0762d0 0x7f728c078790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7298033000 tx=0x7f7298004730 comp rx=0 tx=0).stop 2024-07-26T06:14:33.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 msgr2=0x7f72b0151840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:33.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 0x7f72b0151840 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f72a000d950 tx=0x7f72a000de10 comp rx=0 tx=0).stop 2024-07-26T06:14:33.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 shutdown_connections 2024-07-26T06:14:33.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f728c0762d0 0x7f728c078790 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72b015db10 0x7f72b0152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72b00b3bd0 0x7f72b01498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 --2- 172.21.15.163:0/2612519139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72b00ae870 0x7f72b0151840 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:33.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 >> 172.21.15.163:0/2612519139 conn(0x7f72b00b0e30 msgr2=0x7f72b0156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:33.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 shutdown_connections 2024-07-26T06:14:33.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:33.945+0000 7f72b5600640 1 -- 172.21.15.163:0/2612519139 wait complete. 2024-07-26T06:14:33.970 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:14:33.970 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:34.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b4400640 1 -- 172.21.15.163:0/1812212616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4940050a0 msgr2=0x7fd49400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b4400640 1 --2- 172.21.15.163:0/1812212616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4940050a0 0x7fd49400b9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4006fd0 tx=0x7fd494005830 comp rx=0 tx=0).stop 2024-07-26T06:14:34.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b2400640 1 -- 172.21.15.163:0/1812212616 reap_dead start 2024-07-26T06:14:34.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b4400640 1 -- 172.21.15.163:0/1812212616 shutdown_connections 2024-07-26T06:14:34.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b4400640 1 -- 172.21.15.163:0/1812212616 >> 172.21.15.163:0/1812212616 conn(0x7fd4ac09c7e0 msgr2=0x7fd4ac08ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:34.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b4400640 1 -- 172.21.15.163:0/1812212616 shutdown_connections 2024-07-26T06:14:34.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.185+0000 7fd4b4400640 1 -- 172.21.15.163:0/1812212616 wait complete. 2024-07-26T06:14:34.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.187+0000 7fd4b4400640 1 Processor -- start 2024-07-26T06:14:34.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.187+0000 7fd4b4400640 1 -- start start 2024-07-26T06:14:34.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b4400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4940050a0 0x7fd4ac154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b4400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4ac0ca540 0x7fd4ac0c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b4400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 0x7fd4ac0b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b4400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac0d3570 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b4400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac0d33f0 con 0x7fd4ac0ca540 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b4400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ac0b4240 con 0x7fd4940050a0 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b2400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 0x7fd4ac0b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b2400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 0x7fd4ac0b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49252/0 (socket says 172.21.15.163:49252) 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b2400640 1 -- 172.21.15.163:0/2811800783 learned_addr learned my addr 172.21.15.163:0/2811800783 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b1000640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4ac0ca540 0x7fd4ac0c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.188+0000 7fd4b1a00640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4940050a0 0x7fd4ac154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b2400640 1 -- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4940050a0 msgr2=0x7fd4ac154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b2400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4940050a0 0x7fd4ac154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b2400640 1 -- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4ac0ca540 msgr2=0x7fd4ac0c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b2400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4ac0ca540 0x7fd4ac0c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b2400640 1 -- 172.21.15.163:0/2811800783 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac0caa20 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b1000640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4ac0ca540 0x7fd4ac0c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b2400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 0x7fd4ac0b1010 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400c3a0 tx=0x7fd4a402cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4031050 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4a4006070 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ac0cad00 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4006ca0 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.189+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ac0cb2c0 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.190+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ac0ceb50 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.192+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd4a4034070 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.193+0000 7fd4a9600640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd48c076300 0x7fd48c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.193+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fd4a40b6f70 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.193+0000 7fd4b1a00640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd48c076300 0x7fd48c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.194+0000 7fd4b1a00640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd48c076300 0x7fd48c0787c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006fd0 tx=0x7fd4a0006e40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:34.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.197+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd4a4045030 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.429+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4ac0cb4c0 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.431+0000 7fd4a9600640 1 -- 172.21.15.163:0/2811800783 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5229 (secure 0 0 0) 0x7fd4a4080840 con 0x7fd4ac0b0b60 2024-07-26T06:14:34.432 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:34.432 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":16,"btime":"2024-07-26T06:14:33:929585+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:14:30.337611+0000","modified":"2024-07-26T06:14:33.428686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.163:6835/941040757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":941040757},{"type":"v1","addr":"172.21.15.163:6835","nonce":941040757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-07-26T06:14:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 16 2024-07-26T06:14:34.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd48c076300 msgr2=0x7fd48c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd48c076300 0x7fd48c0787c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006fd0 tx=0x7fd4a0006e40 comp rx=0 tx=0).stop 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 msgr2=0x7fd4ac0b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 0x7fd4ac0b1010 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400c3a0 tx=0x7fd4a402cc40 comp rx=0 tx=0).stop 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 shutdown_connections 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd48c076300 0x7fd48c0787c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd4ac0b0b60 0x7fd4ac0b1010 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4ac0ca540 0x7fd4ac0c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 --2- 172.21.15.163:0/2811800783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4940050a0 0x7fd4ac154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.433+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 >> 172.21.15.163:0/2811800783 conn(0x7fd4ac09c7e0 msgr2=0x7fd4ac0b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.434+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 shutdown_connections 2024-07-26T06:14:34.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.434+0000 7fd4b4400640 1 -- 172.21.15.163:0/2811800783 wait complete. 2024-07-26T06:14:34.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.647+0000 7f7d0e200640 1 -- 172.21.15.163:0/2344845189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 msgr2=0x7f7d080b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.647+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2344845189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d080b2130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7d080b1e30 tx=0x7f7d0002aa30 comp rx=0 tx=0).stop 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.647+0000 7f7d0e200640 1 -- 172.21.15.163:0/2344845189 shutdown_connections 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.647+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2344845189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d080b2130 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.647+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2344845189 >> v2:172.21.15.164:3300/0 conn(0x7f7d08159370 0x7f7d08159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.647+0000 7f7d0e200640 1 -- 172.21.15.163:0/2344845189 >> 172.21.15.163:0/2344845189 conn(0x7f7d080b0e30 msgr2=0x7f7d080b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.648+0000 7f7d0e200640 1 -- 172.21.15.163:0/2344845189 shutdown_connections 2024-07-26T06:14:34.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.648+0000 7f7d0e200640 1 -- 172.21.15.163:0/2344845189 wait complete. 2024-07-26T06:14:34.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.650+0000 7f7d0e200640 1 Processor -- start 2024-07-26T06:14:34.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.650+0000 7f7d0e200640 1 -- start start 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d081561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d08159370 0x7f7d0815df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d08071860 0x7f7d08071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0806f2f0 con 0x7f7d080b4420 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7d080cd210 con 0x7f7d08071860 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0806f470 con 0x7f7d08159370 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0ce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d081561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0ce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d081561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49266/0 (socket says 172.21.15.163:49266) 2024-07-26T06:14:34.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.651+0000 7f7d0ce00640 1 -- 172.21.15.163:0/2668423087 learned_addr learned my addr 172.21.15.163:0/2668423087 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0d800640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d08071860 0x7f7d08071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0ce00640 1 -- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d08159370 msgr2=0x7f7d0815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d07e00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d08159370 0x7f7d0815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0ce00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d08159370 0x7f7d0815df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0ce00640 1 -- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d08071860 msgr2=0x7f7d08071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0ce00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d08071860 0x7f7d08071cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0ce00640 1 -- 172.21.15.163:0/2668423087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d080722e0 con 0x7f7d080b4420 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0d800640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d08071860 0x7f7d08071cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:34.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0ce00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d081561d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00c9a0 tx=0x7f7cfc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:34.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc007bb0 con 0x7f7d080b4420 2024-07-26T06:14:34.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cfc007e40 con 0x7f7d080b4420 2024-07-26T06:14:34.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc005280 con 0x7f7d080b4420 2024-07-26T06:14:34.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d080725c0 con 0x7f7d080b4420 2024-07-26T06:14:34.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.652+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d08152030 con 0x7f7d080b4420 2024-07-26T06:14:34.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.653+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d08072800 con 0x7f7d080b4420 2024-07-26T06:14:34.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.655+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7cfc005420 con 0x7f7d080b4420 2024-07-26T06:14:34.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.656+0000 7f7d04c00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ce80762d0 0x7f7ce8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:34.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.656+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f7cfc097b90 con 0x7f7d080b4420 2024-07-26T06:14:34.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.657+0000 7f7d07e00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ce80762d0 0x7f7ce8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:34.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.658+0000 7f7d07e00640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ce80762d0 0x7f7ce8078790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7d080ae660 tx=0x7f7cf4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:34.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.662+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d08072800 con 0x7f7d080b4420 2024-07-26T06:14:34.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.867+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d080b2130 con 0x7f7d080b4420 2024-07-26T06:14:34.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.870+0000 7f7d04c00640 1 -- 172.21.15.163:0/2668423087 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7f7cfc014030 con 0x7f7d080b4420 2024-07-26T06:14:34.871 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:34.872 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":34,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:31.905523+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:14:28.059274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-07-26T06:14:29.335706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:34.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ce80762d0 msgr2=0x7f7ce8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ce80762d0 0x7f7ce8078790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7d080ae660 tx=0x7f7cf4009040 comp rx=0 tx=0).stop 2024-07-26T06:14:34.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 msgr2=0x7f7d081561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d081561d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00c9a0 tx=0x7f7cfc00ce60 comp rx=0 tx=0).stop 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 shutdown_connections 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ce80762d0 0x7f7ce8078790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d08071860 0x7f7d08071cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d08159370 0x7f7d0815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 --2- 172.21.15.163:0/2668423087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d080b4420 0x7f7d081561d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.873+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 >> 172.21.15.163:0/2668423087 conn(0x7f7d080b0e30 msgr2=0x7f7d080aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.874+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 shutdown_connections 2024-07-26T06:14:34.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:34.874+0000 7f7d0e200640 1 -- 172.21.15.163:0/2668423087 wait complete. 2024-07-26T06:14:34.891 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:14:34.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.089+0000 7fe8d7e00640 1 -- 172.21.15.163:0/3245206921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 msgr2=0x7fe8b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.089+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/3245206921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8b80051c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e50 tx=0x7fe8b8005830 comp rx=0 tx=0).stop 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.089+0000 7fe8d5e00640 1 -- 172.21.15.163:0/3245206921 reap_dead start 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.090+0000 7fe8d7e00640 1 -- 172.21.15.163:0/3245206921 shutdown_connections 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.090+0000 7fe8d7e00640 1 -- 172.21.15.163:0/3245206921 >> 172.21.15.163:0/3245206921 conn(0x7fe8d006d420 msgr2=0x7fe8d006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.090+0000 7fe8d7e00640 1 -- 172.21.15.163:0/3245206921 shutdown_connections 2024-07-26T06:14:35.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.090+0000 7fe8d7e00640 1 -- 172.21.15.163:0/3245206921 wait complete. 2024-07-26T06:14:35.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.091+0000 7fe8d7e00640 1 Processor -- start 2024-07-26T06:14:35.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 -- start start 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8d0152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8d015b6a0 0x7fe8d015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8d01573e0 0x7fe8d0157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d00cc8e0 con 0x7fe8b8004d40 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d006f640 con 0x7fe8d01573e0 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.092+0000 7fe8d7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d00cca60 con 0x7fe8d015b6a0 2024-07-26T06:14:35.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8d0152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8d0152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40478/0 (socket says 172.21.15.163:40478) 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 -- 172.21.15.163:0/1284463248 learned_addr learned my addr 172.21.15.163:0/1284463248 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 -- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8d015b6a0 msgr2=0x7fe8d015bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8d015b6a0 0x7fe8d015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 -- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8d01573e0 msgr2=0x7fe8d0157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8d01573e0 0x7fe8d0157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.093+0000 7fe8d5400640 1 -- 172.21.15.163:0/1284463248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d0156380 con 0x7fe8b8004d40 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.094+0000 7fe8d5400640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8d0152810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40109a0 tx=0x7fe8c4010e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.094+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c400ed20 con 0x7fe8b8004d40 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.094+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8c4004030 con 0x7fe8b8004d40 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.094+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4004460 con 0x7fe8b8004d40 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.094+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d0156620 con 0x7fe8b8004d40 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.094+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d0156be0 con 0x7fe8b8004d40 2024-07-26T06:14:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.095+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d0159760 con 0x7fe8b8004d40 2024-07-26T06:14:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.096+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe8c40035f0 con 0x7fe8b8004d40 2024-07-26T06:14:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.097+0000 7fe8cd600640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe8b0076300 0x7fe8b00787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.097+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fe8c4098480 con 0x7fe8b8004d40 2024-07-26T06:14:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.098+0000 7fe8d4a00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe8b0076300 0x7fe8b00787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:35.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.099+0000 7fe8d4a00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe8b0076300 0x7fe8b00787c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc000fb0 tx=0x7fe8bc001620 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:35.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.101+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe8c4061e70 con 0x7fe8b8004d40 2024-07-26T06:14:35.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.290+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8d015da00 con 0x7fe8b0076300 2024-07-26T06:14:35.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.304+0000 7fe8cd600640 1 -- 172.21.15.163:0/1284463248 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251885 (secure 0 0 0) 0x7fe8d015da00 con 0x7fe8b0076300 2024-07-26T06:14:35.308 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:35.311 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:14:35.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.310+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe8b0076300 msgr2=0x7fe8b00787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.310+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe8b0076300 0x7fe8b00787c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc000fb0 tx=0x7fe8bc001620 comp rx=0 tx=0).stop 2024-07-26T06:14:35.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.310+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 msgr2=0x7fe8d0152810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.310+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8d0152810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40109a0 tx=0x7fe8c4010e60 comp rx=0 tx=0).stop 2024-07-26T06:14:35.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 shutdown_connections 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe8b0076300 0x7fe8b00787c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8d01573e0 0x7fe8d0157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8d015b6a0 0x7fe8d015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 --2- 172.21.15.163:0/1284463248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8b8004d40 0x7fe8d0152810 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 >> 172.21.15.163:0/1284463248 conn(0x7fe8d006d420 msgr2=0x7fe8d0154740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 shutdown_connections 2024-07-26T06:14:35.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.311+0000 7fe8d7e00640 1 -- 172.21.15.163:0/1284463248 wait complete. 2024-07-26T06:14:35.331 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-07-26T06:14:33.664219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220252,"kb_used_data":4044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754468,"statfs":{"total":773094113280,"available":772868575232,"internally_reserved":0,"allocated":4141056,"data_stored":2306851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.807389"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255593+0000","last_change":"2024-07-26T06:14:30.937043+0000","last_active":"2024-07-26T06:14:32.255593+0000","last_peered":"2024-07-26T06:14:32.255593+0000","last_clean":"2024-07-26T06:14:32.255593+0000","last_became_active":"2024-07-26T06:14:30.936733+0000","last_became_peered":"2024-07-26T06:14:30.936733+0000","last_unstale":"2024-07-26T06:14:32.255593+0000","last_undegraded":"2024-07-26T06:14:32.255593+0000","last_fullsized":"2024-07-26T06:14:32.255593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:16.720050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.930040+0000","last_change":"2024-07-26T06:14:30.914532+0000","last_active":"2024-07-26T06:14:30.930040+0000","last_peered":"2024-07-26T06:14:30.930040+0000","last_clean":"2024-07-26T06:14:30.930040+0000","last_became_active":"2024-07-26T06:14:30.913762+0000","last_became_peered":"2024-07-26T06:14:30.913762+0000","last_unstale":"2024-07-26T06:14:30.930040+0000","last_undegraded":"2024-07-26T06:14:30.930040+0000","last_fullsized":"2024-07-26T06:14:30.930040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:40.681640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941895+0000","last_change":"2024-07-26T06:14:30.936293+0000","last_active":"2024-07-26T06:14:30.941895+0000","last_peered":"2024-07-26T06:14:30.941895+0000","last_clean":"2024-07-26T06:14:30.941895+0000","last_became_active":"2024-07-26T06:14:30.935961+0000","last_became_peered":"2024-07-26T06:14:30.935961+0000","last_unstale":"2024-07-26T06:14:30.941895+0000","last_undegraded":"2024-07-26T06:14:30.941895+0000","last_fullsized":"2024-07-26T06:14:30.941895+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:45.290332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939727+0000","last_change":"2024-07-26T06:14:30.935750+0000","last_active":"2024-07-26T06:14:30.939727+0000","last_peered":"2024-07-26T06:14:30.939727+0000","last_clean":"2024-07-26T06:14:30.939727+0000","last_became_active":"2024-07-26T06:14:30.935233+0000","last_became_peered":"2024-07-26T06:14:30.935233+0000","last_unstale":"2024-07-26T06:14:30.939727+0000","last_undegraded":"2024-07-26T06:14:30.939727+0000","last_fullsized":"2024-07-26T06:14:30.939727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:07:02.719589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.918950+0000","last_change":"2024-07-26T06:14:30.909205+0000","last_active":"2024-07-26T06:14:31.918950+0000","last_peered":"2024-07-26T06:14:31.918950+0000","last_clean":"2024-07-26T06:14:31.918950+0000","last_became_active":"2024-07-26T06:14:30.908865+0000","last_became_peered":"2024-07-26T06:14:30.908865+0000","last_unstale":"2024-07-26T06:14:31.918950+0000","last_undegraded":"2024-07-26T06:14:31.918950+0000","last_fullsized":"2024-07-26T06:14:31.918950+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:40.535438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943502+0000","last_change":"2024-07-26T06:14:30.936778+0000","last_active":"2024-07-26T06:14:30.943502+0000","last_peered":"2024-07-26T06:14:30.943502+0000","last_clean":"2024-07-26T06:14:30.943502+0000","last_became_active":"2024-07-26T06:14:30.936465+0000","last_became_peered":"2024-07-26T06:14:30.936465+0000","last_unstale":"2024-07-26T06:14:30.943502+0000","last_undegraded":"2024-07-26T06:14:30.943502+0000","last_fullsized":"2024-07-26T06:14:30.943502+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:27:46.459498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.947216+0000","last_change":"2024-07-26T06:14:30.945157+0000","last_active":"2024-07-26T06:14:30.947216+0000","last_peered":"2024-07-26T06:14:30.947216+0000","last_clean":"2024-07-26T06:14:30.947216+0000","last_became_active":"2024-07-26T06:14:30.944861+0000","last_became_peered":"2024-07-26T06:14:30.944861+0000","last_unstale":"2024-07-26T06:14:30.947216+0000","last_undegraded":"2024-07-26T06:14:30.947216+0000","last_fullsized":"2024-07-26T06:14:30.947216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:46.815340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.936562+0000","last_change":"2024-07-26T06:14:30.928891+0000","last_active":"2024-07-26T06:14:30.936562+0000","last_peered":"2024-07-26T06:14:30.936562+0000","last_clean":"2024-07-26T06:14:30.936562+0000","last_became_active":"2024-07-26T06:14:30.928531+0000","last_became_peered":"2024-07-26T06:14:30.928531+0000","last_unstale":"2024-07-26T06:14:30.936562+0000","last_undegraded":"2024-07-26T06:14:30.936562+0000","last_fullsized":"2024-07-26T06:14:30.936562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:15:45.765158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.938826+0000","last_change":"2024-07-26T06:14:30.922008+0000","last_active":"2024-07-26T06:14:30.938826+0000","last_peered":"2024-07-26T06:14:30.938826+0000","last_clean":"2024-07-26T06:14:30.938826+0000","last_became_active":"2024-07-26T06:14:30.921712+0000","last_became_peered":"2024-07-26T06:14:30.921712+0000","last_unstale":"2024-07-26T06:14:30.938826+0000","last_undegraded":"2024-07-26T06:14:30.938826+0000","last_fullsized":"2024-07-26T06:14:30.938826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:56:58.032909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939310+0000","last_change":"2024-07-26T06:14:30.931244+0000","last_active":"2024-07-26T06:14:30.939310+0000","last_peered":"2024-07-26T06:14:30.939310+0000","last_clean":"2024-07-26T06:14:30.939310+0000","last_became_active":"2024-07-26T06:14:30.930910+0000","last_became_peered":"2024-07-26T06:14:30.930910+0000","last_unstale":"2024-07-26T06:14:30.939310+0000","last_undegraded":"2024-07-26T06:14:30.939310+0000","last_fullsized":"2024-07-26T06:14:30.939310+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:56:58.032909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942546+0000","last_change":"2024-07-26T06:14:30.934242+0000","last_active":"2024-07-26T06:14:30.942546+0000","last_peered":"2024-07-26T06:14:30.942546+0000","last_clean":"2024-07-26T06:14:30.942546+0000","last_became_active":"2024-07-26T06:14:30.933730+0000","last_became_peered":"2024-07-26T06:14:30.933730+0000","last_unstale":"2024-07-26T06:14:30.942546+0000","last_undegraded":"2024-07-26T06:14:30.942546+0000","last_fullsized":"2024-07-26T06:14:30.942546+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:47:58.971707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942101+0000","last_change":"2024-07-26T06:14:30.935656+0000","last_active":"2024-07-26T06:14:30.942101+0000","last_peered":"2024-07-26T06:14:30.942101+0000","last_clean":"2024-07-26T06:14:30.942101+0000","last_became_active":"2024-07-26T06:14:30.935154+0000","last_became_peered":"2024-07-26T06:14:30.935154+0000","last_unstale":"2024-07-26T06:14:30.942101+0000","last_undegraded":"2024-07-26T06:14:30.942101+0000","last_fullsized":"2024-07-26T06:14:30.942101+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:30:47.461579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919091+0000","last_change":"2024-07-26T06:14:30.915521+0000","last_active":"2024-07-26T06:14:31.919091+0000","last_peered":"2024-07-26T06:14:31.919091+0000","last_clean":"2024-07-26T06:14:31.919091+0000","last_became_active":"2024-07-26T06:14:30.914996+0000","last_became_peered":"2024-07-26T06:14:30.914996+0000","last_unstale":"2024-07-26T06:14:31.919091+0000","last_undegraded":"2024-07-26T06:14:31.919091+0000","last_fullsized":"2024-07-26T06:14:31.919091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:19.392502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.932744+0000","last_change":"2024-07-26T06:14:30.917397+0000","last_active":"2024-07-26T06:14:30.932744+0000","last_peered":"2024-07-26T06:14:30.932744+0000","last_clean":"2024-07-26T06:14:30.932744+0000","last_became_active":"2024-07-26T06:14:30.917113+0000","last_became_peered":"2024-07-26T06:14:30.917113+0000","last_unstale":"2024-07-26T06:14:30.932744+0000","last_undegraded":"2024-07-26T06:14:30.932744+0000","last_fullsized":"2024-07-26T06:14:30.932744+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:57.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941278+0000","last_change":"2024-07-26T06:14:30.932610+0000","last_active":"2024-07-26T06:14:30.941278+0000","last_peered":"2024-07-26T06:14:30.941278+0000","last_clean":"2024-07-26T06:14:30.941278+0000","last_became_active":"2024-07-26T06:14:30.931759+0000","last_became_peered":"2024-07-26T06:14:30.931759+0000","last_unstale":"2024-07-26T06:14:30.941278+0000","last_undegraded":"2024-07-26T06:14:30.941278+0000","last_fullsized":"2024-07-26T06:14:30.941278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:28:27.267772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946859+0000","last_change":"2024-07-26T06:14:30.943069+0000","last_active":"2024-07-26T06:14:30.946859+0000","last_peered":"2024-07-26T06:14:30.946859+0000","last_clean":"2024-07-26T06:14:30.946859+0000","last_became_active":"2024-07-26T06:14:30.942190+0000","last_became_peered":"2024-07-26T06:14:30.942190+0000","last_unstale":"2024-07-26T06:14:30.946859+0000","last_undegraded":"2024-07-26T06:14:30.946859+0000","last_fullsized":"2024-07-26T06:14:30.946859+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:45.290332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944656+0000","last_change":"2024-07-26T06:14:30.938725+0000","last_active":"2024-07-26T06:14:30.944656+0000","last_peered":"2024-07-26T06:14:30.944656+0000","last_clean":"2024-07-26T06:14:30.944656+0000","last_became_active":"2024-07-26T06:14:30.938420+0000","last_became_peered":"2024-07-26T06:14:30.938420+0000","last_unstale":"2024-07-26T06:14:30.944656+0000","last_undegraded":"2024-07-26T06:14:30.944656+0000","last_fullsized":"2024-07-26T06:14:30.944656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:16.720050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.932783+0000","last_change":"2024-07-26T06:14:30.918857+0000","last_active":"2024-07-26T06:14:30.932783+0000","last_peered":"2024-07-26T06:14:30.932783+0000","last_clean":"2024-07-26T06:14:30.932783+0000","last_became_active":"2024-07-26T06:14:30.918601+0000","last_became_peered":"2024-07-26T06:14:30.918601+0000","last_unstale":"2024-07-26T06:14:30.932783+0000","last_undegraded":"2024-07-26T06:14:30.932783+0000","last_fullsized":"2024-07-26T06:14:30.932783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:45:03.127108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.933342+0000","last_change":"2024-07-26T06:14:30.923161+0000","last_active":"2024-07-26T06:14:30.933342+0000","last_peered":"2024-07-26T06:14:30.933342+0000","last_clean":"2024-07-26T06:14:30.933342+0000","last_became_active":"2024-07-26T06:14:30.922893+0000","last_became_peered":"2024-07-26T06:14:30.922893+0000","last_unstale":"2024-07-26T06:14:30.933342+0000","last_undegraded":"2024-07-26T06:14:30.933342+0000","last_fullsized":"2024-07-26T06:14:30.933342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:12:49.364457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919057+0000","last_change":"2024-07-26T06:14:30.939350+0000","last_active":"2024-07-26T06:14:31.919057+0000","last_peered":"2024-07-26T06:14:31.919057+0000","last_clean":"2024-07-26T06:14:31.919057+0000","last_became_active":"2024-07-26T06:14:30.938968+0000","last_became_peered":"2024-07-26T06:14:30.938968+0000","last_unstale":"2024-07-26T06:14:31.919057+0000","last_undegraded":"2024-07-26T06:14:31.919057+0000","last_fullsized":"2024-07-26T06:14:31.919057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:06.781871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.930216+0000","last_change":"2024-07-26T06:14:30.919374+0000","last_active":"2024-07-26T06:14:30.930216+0000","last_peered":"2024-07-26T06:14:30.930216+0000","last_clean":"2024-07-26T06:14:30.930216+0000","last_became_active":"2024-07-26T06:14:30.918846+0000","last_became_peered":"2024-07-26T06:14:30.918846+0000","last_unstale":"2024-07-26T06:14:30.930216+0000","last_undegraded":"2024-07-26T06:14:30.930216+0000","last_fullsized":"2024-07-26T06:14:30.930216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:48:13.625747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940323+0000","last_change":"2024-07-26T06:14:30.932146+0000","last_active":"2024-07-26T06:14:30.940323+0000","last_peered":"2024-07-26T06:14:30.940323+0000","last_clean":"2024-07-26T06:14:30.940323+0000","last_became_active":"2024-07-26T06:14:30.931859+0000","last_became_peered":"2024-07-26T06:14:30.931859+0000","last_unstale":"2024-07-26T06:14:30.940323+0000","last_undegraded":"2024-07-26T06:14:30.940323+0000","last_fullsized":"2024-07-26T06:14:30.940323+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:21:47.556217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.937561+0000","last_change":"2024-07-26T06:14:30.929342+0000","last_active":"2024-07-26T06:14:30.937561+0000","last_peered":"2024-07-26T06:14:30.937561+0000","last_clean":"2024-07-26T06:14:30.937561+0000","last_became_active":"2024-07-26T06:14:30.929046+0000","last_became_peered":"2024-07-26T06:14:30.929046+0000","last_unstale":"2024-07-26T06:14:30.937561+0000","last_undegraded":"2024-07-26T06:14:30.937561+0000","last_fullsized":"2024-07-26T06:14:30.937561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:54.664264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941035+0000","last_change":"2024-07-26T06:14:30.928572+0000","last_active":"2024-07-26T06:14:30.941035+0000","last_peered":"2024-07-26T06:14:30.941035+0000","last_clean":"2024-07-26T06:14:30.941035+0000","last_became_active":"2024-07-26T06:14:30.928246+0000","last_became_peered":"2024-07-26T06:14:30.928246+0000","last_unstale":"2024-07-26T06:14:30.941035+0000","last_undegraded":"2024-07-26T06:14:30.941035+0000","last_fullsized":"2024-07-26T06:14:30.941035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:10:19.889725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939643+0000","last_change":"2024-07-26T06:14:30.932616+0000","last_active":"2024-07-26T06:14:30.939643+0000","last_peered":"2024-07-26T06:14:30.939643+0000","last_clean":"2024-07-26T06:14:30.939643+0000","last_became_active":"2024-07-26T06:14:30.932352+0000","last_became_peered":"2024-07-26T06:14:30.932352+0000","last_unstale":"2024-07-26T06:14:30.939643+0000","last_undegraded":"2024-07-26T06:14:30.939643+0000","last_fullsized":"2024-07-26T06:14:30.939643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:09.397816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.934120+0000","last_change":"2024-07-26T06:14:30.930180+0000","last_active":"2024-07-26T06:14:30.934120+0000","last_peered":"2024-07-26T06:14:30.934120+0000","last_clean":"2024-07-26T06:14:30.934120+0000","last_became_active":"2024-07-26T06:14:30.929873+0000","last_became_peered":"2024-07-26T06:14:30.929873+0000","last_unstale":"2024-07-26T06:14:30.934120+0000","last_undegraded":"2024-07-26T06:14:30.934120+0000","last_fullsized":"2024-07-26T06:14:30.934120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:22.448217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939828+0000","last_change":"2024-07-26T06:14:30.936704+0000","last_active":"2024-07-26T06:14:30.939828+0000","last_peered":"2024-07-26T06:14:30.939828+0000","last_clean":"2024-07-26T06:14:30.939828+0000","last_became_active":"2024-07-26T06:14:30.936074+0000","last_became_peered":"2024-07-26T06:14:30.936074+0000","last_unstale":"2024-07-26T06:14:30.939828+0000","last_undegraded":"2024-07-26T06:14:30.939828+0000","last_fullsized":"2024-07-26T06:14:30.939828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:37:38.304407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940786+0000","last_change":"2024-07-26T06:14:30.930555+0000","last_active":"2024-07-26T06:14:30.940786+0000","last_peered":"2024-07-26T06:14:30.940786+0000","last_clean":"2024-07-26T06:14:30.940786+0000","last_became_active":"2024-07-26T06:14:30.930226+0000","last_became_peered":"2024-07-26T06:14:30.930226+0000","last_unstale":"2024-07-26T06:14:30.940786+0000","last_undegraded":"2024-07-26T06:14:30.940786+0000","last_fullsized":"2024-07-26T06:14:30.940786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:27:38.310414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.929573+0000","last_change":"2024-07-26T06:14:30.918965+0000","last_active":"2024-07-26T06:14:30.929573+0000","last_peered":"2024-07-26T06:14:30.929573+0000","last_clean":"2024-07-26T06:14:30.929573+0000","last_became_active":"2024-07-26T06:14:30.918648+0000","last_became_peered":"2024-07-26T06:14:30.918648+0000","last_unstale":"2024-07-26T06:14:30.929573+0000","last_undegraded":"2024-07-26T06:14:30.929573+0000","last_fullsized":"2024-07-26T06:14:30.929573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:24.078491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942168+0000","last_change":"2024-07-26T06:14:30.929862+0000","last_active":"2024-07-26T06:14:30.942168+0000","last_peered":"2024-07-26T06:14:30.942168+0000","last_clean":"2024-07-26T06:14:30.942168+0000","last_became_active":"2024-07-26T06:14:30.929588+0000","last_became_peered":"2024-07-26T06:14:30.929588+0000","last_unstale":"2024-07-26T06:14:30.942168+0000","last_undegraded":"2024-07-26T06:14:30.942168+0000","last_fullsized":"2024-07-26T06:14:30.942168+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:10:19.889725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255203+0000","last_change":"2024-07-26T06:14:30.922576+0000","last_active":"2024-07-26T06:14:32.255203+0000","last_peered":"2024-07-26T06:14:32.255203+0000","last_clean":"2024-07-26T06:14:32.255203+0000","last_became_active":"2024-07-26T06:14:30.922225+0000","last_became_peered":"2024-07-26T06:14:30.922225+0000","last_unstale":"2024-07-26T06:14:32.255203+0000","last_undegraded":"2024-07-26T06:14:32.255203+0000","last_fullsized":"2024-07-26T06:14:32.255203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:27:38.310414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946840+0000","last_change":"2024-07-26T06:14:30.942849+0000","last_active":"2024-07-26T06:14:30.946840+0000","last_peered":"2024-07-26T06:14:30.946840+0000","last_clean":"2024-07-26T06:14:30.946840+0000","last_became_active":"2024-07-26T06:14:30.941822+0000","last_became_peered":"2024-07-26T06:14:30.941822+0000","last_unstale":"2024-07-26T06:14:30.946840+0000","last_undegraded":"2024-07-26T06:14:30.946840+0000","last_fullsized":"2024-07-26T06:14:30.946840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:03.697326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941404+0000","last_change":"2024-07-26T06:14:30.936955+0000","last_active":"2024-07-26T06:14:30.941404+0000","last_peered":"2024-07-26T06:14:30.941404+0000","last_clean":"2024-07-26T06:14:30.941404+0000","last_became_active":"2024-07-26T06:14:30.936554+0000","last_became_peered":"2024-07-26T06:14:30.936554+0000","last_unstale":"2024-07-26T06:14:30.941404+0000","last_undegraded":"2024-07-26T06:14:30.941404+0000","last_fullsized":"2024-07-26T06:14:30.941404+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:15:45.765158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255694+0000","last_change":"2024-07-26T06:14:29.904778+0000","last_active":"2024-07-26T06:14:32.255694+0000","last_peered":"2024-07-26T06:14:32.255694+0000","last_clean":"2024-07-26T06:14:32.255694+0000","last_became_active":"2024-07-26T06:14:29.904495+0000","last_became_peered":"2024-07-26T06:14:29.904495+0000","last_unstale":"2024-07-26T06:14:32.255694+0000","last_undegraded":"2024-07-26T06:14:32.255694+0000","last_fullsized":"2024-07-26T06:14:32.255694+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:18.381616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255538+0000","last_change":"2024-07-26T06:14:30.940539+0000","last_active":"2024-07-26T06:14:32.255538+0000","last_peered":"2024-07-26T06:14:32.255538+0000","last_clean":"2024-07-26T06:14:32.255538+0000","last_became_active":"2024-07-26T06:14:30.939690+0000","last_became_peered":"2024-07-26T06:14:30.939690+0000","last_unstale":"2024-07-26T06:14:32.255538+0000","last_undegraded":"2024-07-26T06:14:32.255538+0000","last_fullsized":"2024-07-26T06:14:32.255538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:28:27.267772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.936797+0000","last_change":"2024-07-26T06:14:29.902273+0000","last_active":"2024-07-26T06:14:30.936797+0000","last_peered":"2024-07-26T06:14:30.936797+0000","last_clean":"2024-07-26T06:14:30.936797+0000","last_became_active":"2024-07-26T06:14:29.901774+0000","last_became_peered":"2024-07-26T06:14:29.901774+0000","last_unstale":"2024-07-26T06:14:30.936797+0000","last_undegraded":"2024-07-26T06:14:30.936797+0000","last_fullsized":"2024-07-26T06:14:30.936797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:05.770984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.931569+0000","last_change":"2024-07-26T06:14:30.918223+0000","last_active":"2024-07-26T06:14:30.931569+0000","last_peered":"2024-07-26T06:14:30.931569+0000","last_clean":"2024-07-26T06:14:30.931569+0000","last_became_active":"2024-07-26T06:14:30.917834+0000","last_became_peered":"2024-07-26T06:14:30.917834+0000","last_unstale":"2024-07-26T06:14:30.931569+0000","last_undegraded":"2024-07-26T06:14:30.931569+0000","last_fullsized":"2024-07-26T06:14:30.931569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:06:12.036616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.934190+0000","last_change":"2024-07-26T06:14:29.887831+0000","last_active":"2024-07-26T06:14:30.934190+0000","last_peered":"2024-07-26T06:14:30.934190+0000","last_clean":"2024-07-26T06:14:30.934190+0000","last_became_active":"2024-07-26T06:14:29.887096+0000","last_became_peered":"2024-07-26T06:14:29.887096+0000","last_unstale":"2024-07-26T06:14:30.934190+0000","last_undegraded":"2024-07-26T06:14:30.934190+0000","last_fullsized":"2024-07-26T06:14:30.934190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:48:58.632151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939808+0000","last_change":"2024-07-26T06:14:30.936498+0000","last_active":"2024-07-26T06:14:30.939808+0000","last_peered":"2024-07-26T06:14:30.939808+0000","last_clean":"2024-07-26T06:14:30.939808+0000","last_became_active":"2024-07-26T06:14:30.935664+0000","last_became_peered":"2024-07-26T06:14:30.935664+0000","last_unstale":"2024-07-26T06:14:30.939808+0000","last_undegraded":"2024-07-26T06:14:30.939808+0000","last_fullsized":"2024-07-26T06:14:30.939808+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:56:02.763778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919170+0000","last_change":"2024-07-26T06:14:29.892846+0000","last_active":"2024-07-26T06:14:31.919170+0000","last_peered":"2024-07-26T06:14:31.919170+0000","last_clean":"2024-07-26T06:14:31.919170+0000","last_became_active":"2024-07-26T06:14:29.890684+0000","last_became_peered":"2024-07-26T06:14:29.890684+0000","last_unstale":"2024-07-26T06:14:31.919170+0000","last_undegraded":"2024-07-26T06:14:31.919170+0000","last_fullsized":"2024-07-26T06:14:31.919170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:55.950964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944544+0000","last_change":"2024-07-26T06:14:30.939827+0000","last_active":"2024-07-26T06:14:30.944544+0000","last_peered":"2024-07-26T06:14:30.944544+0000","last_clean":"2024-07-26T06:14:30.944544+0000","last_became_active":"2024-07-26T06:14:30.939558+0000","last_became_peered":"2024-07-26T06:14:30.939558+0000","last_unstale":"2024-07-26T06:14:30.944544+0000","last_undegraded":"2024-07-26T06:14:30.944544+0000","last_fullsized":"2024-07-26T06:14:30.944544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:30:47.461579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941682+0000","last_change":"2024-07-26T06:14:29.886269+0000","last_active":"2024-07-26T06:14:30.941682+0000","last_peered":"2024-07-26T06:14:30.941682+0000","last_clean":"2024-07-26T06:14:30.941682+0000","last_became_active":"2024-07-26T06:14:29.885560+0000","last_became_peered":"2024-07-26T06:14:29.885560+0000","last_unstale":"2024-07-26T06:14:30.941682+0000","last_undegraded":"2024-07-26T06:14:30.941682+0000","last_fullsized":"2024-07-26T06:14:30.941682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:27:21.475143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944177+0000","last_change":"2024-07-26T06:14:30.938191+0000","last_active":"2024-07-26T06:14:30.944177+0000","last_peered":"2024-07-26T06:14:30.944177+0000","last_clean":"2024-07-26T06:14:30.944177+0000","last_became_active":"2024-07-26T06:14:30.937813+0000","last_became_peered":"2024-07-26T06:14:30.937813+0000","last_unstale":"2024-07-26T06:14:30.944177+0000","last_undegraded":"2024-07-26T06:14:30.944177+0000","last_fullsized":"2024-07-26T06:14:30.944177+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:46.815340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940310+0000","last_change":"2024-07-26T06:14:29.889694+0000","last_active":"2024-07-26T06:14:30.940310+0000","last_peered":"2024-07-26T06:14:30.940310+0000","last_clean":"2024-07-26T06:14:30.940310+0000","last_became_active":"2024-07-26T06:14:29.889029+0000","last_became_peered":"2024-07-26T06:14:29.889029+0000","last_unstale":"2024-07-26T06:14:30.940310+0000","last_undegraded":"2024-07-26T06:14:30.940310+0000","last_fullsized":"2024-07-26T06:14:30.940310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:28:10.263397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.938025+0000","last_change":"2024-07-26T06:14:30.925886+0000","last_active":"2024-07-26T06:14:30.938025+0000","last_peered":"2024-07-26T06:14:30.938025+0000","last_clean":"2024-07-26T06:14:30.938025+0000","last_became_active":"2024-07-26T06:14:30.925583+0000","last_became_peered":"2024-07-26T06:14:30.925583+0000","last_unstale":"2024-07-26T06:14:30.938025+0000","last_undegraded":"2024-07-26T06:14:30.938025+0000","last_fullsized":"2024-07-26T06:14:30.938025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:09.397816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.906920+0000","last_change":"2024-07-26T06:14:29.891691+0000","last_active":"2024-07-26T06:14:30.906920+0000","last_peered":"2024-07-26T06:14:30.906920+0000","last_clean":"2024-07-26T06:14:30.906920+0000","last_became_active":"2024-07-26T06:14:29.891434+0000","last_became_peered":"2024-07-26T06:14:29.891434+0000","last_unstale":"2024-07-26T06:14:30.906920+0000","last_undegraded":"2024-07-26T06:14:30.906920+0000","last_fullsized":"2024-07-26T06:14:30.906920+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:56:01.752892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939615+0000","last_change":"2024-07-26T06:14:30.934878+0000","last_active":"2024-07-26T06:14:30.939615+0000","last_peered":"2024-07-26T06:14:30.939615+0000","last_clean":"2024-07-26T06:14:30.939615+0000","last_became_active":"2024-07-26T06:14:30.934577+0000","last_became_peered":"2024-07-26T06:14:30.934577+0000","last_unstale":"2024-07-26T06:14:30.939615+0000","last_undegraded":"2024-07-26T06:14:30.939615+0000","last_fullsized":"2024-07-26T06:14:30.939615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:21:47.556217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.905926+0000","last_change":"2024-07-26T06:14:29.899602+0000","last_active":"2024-07-26T06:14:30.905926+0000","last_peered":"2024-07-26T06:14:30.905926+0000","last_clean":"2024-07-26T06:14:30.905926+0000","last_became_active":"2024-07-26T06:14:29.899306+0000","last_became_peered":"2024-07-26T06:14:29.899306+0000","last_unstale":"2024-07-26T06:14:30.905926+0000","last_undegraded":"2024-07-26T06:14:30.905926+0000","last_fullsized":"2024-07-26T06:14:30.905926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:53.699963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941803+0000","last_change":"2024-07-26T06:14:30.934708+0000","last_active":"2024-07-26T06:14:30.941803+0000","last_peered":"2024-07-26T06:14:30.941803+0000","last_clean":"2024-07-26T06:14:30.941803+0000","last_became_active":"2024-07-26T06:14:30.934421+0000","last_became_peered":"2024-07-26T06:14:30.934421+0000","last_unstale":"2024-07-26T06:14:30.941803+0000","last_undegraded":"2024-07-26T06:14:30.941803+0000","last_fullsized":"2024-07-26T06:14:30.941803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:16.720050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.910525+0000","last_change":"2024-07-26T06:14:29.902854+0000","last_active":"2024-07-26T06:14:30.910525+0000","last_peered":"2024-07-26T06:14:30.910525+0000","last_clean":"2024-07-26T06:14:30.910525+0000","last_became_active":"2024-07-26T06:14:29.901454+0000","last_became_peered":"2024-07-26T06:14:29.901454+0000","last_unstale":"2024-07-26T06:14:30.910525+0000","last_undegraded":"2024-07-26T06:14:30.910525+0000","last_fullsized":"2024-07-26T06:14:30.910525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:30.425545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.932701+0000","last_change":"2024-07-26T06:14:30.917577+0000","last_active":"2024-07-26T06:14:30.932701+0000","last_peered":"2024-07-26T06:14:30.932701+0000","last_clean":"2024-07-26T06:14:30.932701+0000","last_became_active":"2024-07-26T06:14:30.917214+0000","last_became_peered":"2024-07-26T06:14:30.917214+0000","last_unstale":"2024-07-26T06:14:30.932701+0000","last_undegraded":"2024-07-26T06:14:30.932701+0000","last_fullsized":"2024-07-26T06:14:30.932701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:53.787034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.903784+0000","last_change":"2024-07-26T06:14:29.899851+0000","last_active":"2024-07-26T06:14:30.903784+0000","last_peered":"2024-07-26T06:14:30.903784+0000","last_clean":"2024-07-26T06:14:30.903784+0000","last_became_active":"2024-07-26T06:14:29.899536+0000","last_became_peered":"2024-07-26T06:14:29.899536+0000","last_unstale":"2024-07-26T06:14:30.903784+0000","last_undegraded":"2024-07-26T06:14:30.903784+0000","last_fullsized":"2024-07-26T06:14:30.903784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:18.692721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939004+0000","last_change":"2024-07-26T06:14:30.934383+0000","last_active":"2024-07-26T06:14:30.939004+0000","last_peered":"2024-07-26T06:14:30.939004+0000","last_clean":"2024-07-26T06:14:30.939004+0000","last_became_active":"2024-07-26T06:14:30.933971+0000","last_became_peered":"2024-07-26T06:14:30.933971+0000","last_unstale":"2024-07-26T06:14:30.939004+0000","last_undegraded":"2024-07-26T06:14:30.939004+0000","last_fullsized":"2024-07-26T06:14:30.939004+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:30:49.519398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943265+0000","last_change":"2024-07-26T06:14:29.902544+0000","last_active":"2024-07-26T06:14:30.943265+0000","last_peered":"2024-07-26T06:14:30.943265+0000","last_clean":"2024-07-26T06:14:30.943265+0000","last_became_active":"2024-07-26T06:14:29.902256+0000","last_became_peered":"2024-07-26T06:14:29.902256+0000","last_unstale":"2024-07-26T06:14:30.943265+0000","last_undegraded":"2024-07-26T06:14:30.943265+0000","last_fullsized":"2024-07-26T06:14:30.943265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:47:57.960821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943919+0000","last_change":"2024-07-26T06:14:30.936996+0000","last_active":"2024-07-26T06:14:30.943919+0000","last_peered":"2024-07-26T06:14:30.943919+0000","last_clean":"2024-07-26T06:14:30.943919+0000","last_became_active":"2024-07-26T06:14:30.935930+0000","last_became_peered":"2024-07-26T06:14:30.935930+0000","last_unstale":"2024-07-26T06:14:30.943919+0000","last_undegraded":"2024-07-26T06:14:30.943919+0000","last_fullsized":"2024-07-26T06:14:30.943919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:03:33.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946561+0000","last_change":"2024-07-26T06:14:29.890840+0000","last_active":"2024-07-26T06:14:30.946561+0000","last_peered":"2024-07-26T06:14:30.946561+0000","last_clean":"2024-07-26T06:14:30.946561+0000","last_became_active":"2024-07-26T06:14:29.886784+0000","last_became_peered":"2024-07-26T06:14:29.886784+0000","last_unstale":"2024-07-26T06:14:30.946561+0000","last_undegraded":"2024-07-26T06:14:30.946561+0000","last_fullsized":"2024-07-26T06:14:30.946561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:32:05.124118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939681+0000","last_change":"2024-07-26T06:14:30.935028+0000","last_active":"2024-07-26T06:14:30.939681+0000","last_peered":"2024-07-26T06:14:30.939681+0000","last_clean":"2024-07-26T06:14:30.939681+0000","last_became_active":"2024-07-26T06:14:30.934647+0000","last_became_peered":"2024-07-26T06:14:30.934647+0000","last_unstale":"2024-07-26T06:14:30.939681+0000","last_undegraded":"2024-07-26T06:14:30.939681+0000","last_fullsized":"2024-07-26T06:14:30.939681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:37:24.270350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944842+0000","last_change":"2024-07-26T06:14:29.899714+0000","last_active":"2024-07-26T06:14:30.944842+0000","last_peered":"2024-07-26T06:14:30.944842+0000","last_clean":"2024-07-26T06:14:30.944842+0000","last_became_active":"2024-07-26T06:14:29.899402+0000","last_became_peered":"2024-07-26T06:14:29.899402+0000","last_unstale":"2024-07-26T06:14:30.944842+0000","last_undegraded":"2024-07-26T06:14:30.944842+0000","last_fullsized":"2024-07-26T06:14:30.944842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:34:06.610898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.933246+0000","last_change":"2024-07-26T06:14:30.919761+0000","last_active":"2024-07-26T06:14:30.933246+0000","last_peered":"2024-07-26T06:14:30.933246+0000","last_clean":"2024-07-26T06:14:30.933246+0000","last_became_active":"2024-07-26T06:14:30.919453+0000","last_became_peered":"2024-07-26T06:14:30.919453+0000","last_unstale":"2024-07-26T06:14:30.933246+0000","last_undegraded":"2024-07-26T06:14:30.933246+0000","last_fullsized":"2024-07-26T06:14:30.933246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:36:53.985771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.913074+0000","last_change":"2024-07-26T06:14:29.894224+0000","last_active":"2024-07-26T06:14:30.913074+0000","last_peered":"2024-07-26T06:14:30.913074+0000","last_clean":"2024-07-26T06:14:30.913074+0000","last_became_active":"2024-07-26T06:14:29.893561+0000","last_became_peered":"2024-07-26T06:14:29.893561+0000","last_unstale":"2024-07-26T06:14:30.913074+0000","last_undegraded":"2024-07-26T06:14:30.913074+0000","last_fullsized":"2024-07-26T06:14:30.913074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:53.699963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946874+0000","last_change":"2024-07-26T06:14:30.943291+0000","last_active":"2024-07-26T06:14:30.946874+0000","last_peered":"2024-07-26T06:14:30.946874+0000","last_clean":"2024-07-26T06:14:30.946874+0000","last_became_active":"2024-07-26T06:14:30.942541+0000","last_became_peered":"2024-07-26T06:14:30.942541+0000","last_unstale":"2024-07-26T06:14:30.946874+0000","last_undegraded":"2024-07-26T06:14:30.946874+0000","last_fullsized":"2024-07-26T06:14:30.946874+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:03:33.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.906732+0000","last_change":"2024-07-26T06:14:29.900504+0000","last_active":"2024-07-26T06:14:30.906732+0000","last_peered":"2024-07-26T06:14:30.906732+0000","last_clean":"2024-07-26T06:14:30.906732+0000","last_became_active":"2024-07-26T06:14:29.900189+0000","last_became_peered":"2024-07-26T06:14:29.900189+0000","last_unstale":"2024-07-26T06:14:30.906732+0000","last_undegraded":"2024-07-26T06:14:30.906732+0000","last_fullsized":"2024-07-26T06:14:30.906732+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:07:01.708703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255421+0000","last_change":"2024-07-26T06:14:30.932244+0000","last_active":"2024-07-26T06:14:32.255421+0000","last_peered":"2024-07-26T06:14:32.255421+0000","last_clean":"2024-07-26T06:14:32.255421+0000","last_became_active":"2024-07-26T06:14:30.930521+0000","last_became_peered":"2024-07-26T06:14:30.930521+0000","last_unstale":"2024-07-26T06:14:32.255421+0000","last_undegraded":"2024-07-26T06:14:32.255421+0000","last_fullsized":"2024-07-26T06:14:32.255421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:47:58.971707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946696+0000","last_change":"2024-07-26T06:14:29.883552+0000","last_active":"2024-07-26T06:14:30.946696+0000","last_peered":"2024-07-26T06:14:30.946696+0000","last_clean":"2024-07-26T06:14:30.946696+0000","last_became_active":"2024-07-26T06:14:29.883251+0000","last_became_peered":"2024-07-26T06:14:29.883251+0000","last_unstale":"2024-07-26T06:14:30.946696+0000","last_undegraded":"2024-07-26T06:14:30.946696+0000","last_fullsized":"2024-07-26T06:14:30.946696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:32.530603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941662+0000","last_change":"2024-07-26T06:14:30.935702+0000","last_active":"2024-07-26T06:14:30.941662+0000","last_peered":"2024-07-26T06:14:30.941662+0000","last_clean":"2024-07-26T06:14:30.941662+0000","last_became_active":"2024-07-26T06:14:30.935398+0000","last_became_peered":"2024-07-26T06:14:30.935398+0000","last_unstale":"2024-07-26T06:14:30.941662+0000","last_undegraded":"2024-07-26T06:14:30.941662+0000","last_fullsized":"2024-07-26T06:14:30.941662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:37:24.270350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.936061+0000","last_change":"2024-07-26T06:14:29.884910+0000","last_active":"2024-07-26T06:14:30.936061+0000","last_peered":"2024-07-26T06:14:30.936061+0000","last_clean":"2024-07-26T06:14:30.936061+0000","last_became_active":"2024-07-26T06:14:29.883104+0000","last_became_peered":"2024-07-26T06:14:29.883104+0000","last_unstale":"2024-07-26T06:14:30.936061+0000","last_undegraded":"2024-07-26T06:14:30.936061+0000","last_fullsized":"2024-07-26T06:14:30.936061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:20:10.690825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.945316+0000","last_change":"2024-07-26T06:14:30.942257+0000","last_active":"2024-07-26T06:14:30.945316+0000","last_peered":"2024-07-26T06:14:30.945316+0000","last_clean":"2024-07-26T06:14:30.945316+0000","last_became_active":"2024-07-26T06:14:30.941542+0000","last_became_peered":"2024-07-26T06:14:30.941542+0000","last_unstale":"2024-07-26T06:14:30.945316+0000","last_undegraded":"2024-07-26T06:14:30.945316+0000","last_fullsized":"2024-07-26T06:14:30.945316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:15.365238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944451+0000","last_change":"2024-07-26T06:14:29.898758+0000","last_active":"2024-07-26T06:14:30.944451+0000","last_peered":"2024-07-26T06:14:30.944451+0000","last_clean":"2024-07-26T06:14:30.944451+0000","last_became_active":"2024-07-26T06:14:29.898498+0000","last_became_peered":"2024-07-26T06:14:29.898498+0000","last_unstale":"2024-07-26T06:14:30.944451+0000","last_undegraded":"2024-07-26T06:14:30.944451+0000","last_fullsized":"2024-07-26T06:14:30.944451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:18.082903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.922014+0000","last_change":"2024-07-26T06:14:30.918622+0000","last_active":"2024-07-26T06:14:30.922014+0000","last_peered":"2024-07-26T06:14:30.922014+0000","last_clean":"2024-07-26T06:14:30.922014+0000","last_became_active":"2024-07-26T06:14:30.918322+0000","last_became_peered":"2024-07-26T06:14:30.918322+0000","last_unstale":"2024-07-26T06:14:30.922014+0000","last_undegraded":"2024-07-26T06:14:30.922014+0000","last_fullsized":"2024-07-26T06:14:30.922014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:48:13.625747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939703+0000","last_change":"2024-07-26T06:14:29.889911+0000","last_active":"2024-07-26T06:14:30.939703+0000","last_peered":"2024-07-26T06:14:30.939703+0000","last_clean":"2024-07-26T06:14:30.939703+0000","last_became_active":"2024-07-26T06:14:29.889021+0000","last_became_peered":"2024-07-26T06:14:29.889021+0000","last_unstale":"2024-07-26T06:14:30.939703+0000","last_undegraded":"2024-07-26T06:14:30.939703+0000","last_fullsized":"2024-07-26T06:14:30.939703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:48.165552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919716+0000","last_change":"2024-07-26T06:14:30.931601+0000","last_active":"2024-07-26T06:14:31.919716+0000","last_peered":"2024-07-26T06:14:31.919716+0000","last_clean":"2024-07-26T06:14:31.919716+0000","last_became_active":"2024-07-26T06:14:30.930659+0000","last_became_peered":"2024-07-26T06:14:30.930659+0000","last_unstale":"2024-07-26T06:14:31.919716+0000","last_undegraded":"2024-07-26T06:14:31.919716+0000","last_fullsized":"2024-07-26T06:14:31.919716+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:54.664264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.901696+0000","last_change":"2024-07-26T06:14:29.891888+0000","last_active":"2024-07-26T06:14:30.901696+0000","last_peered":"2024-07-26T06:14:30.901696+0000","last_clean":"2024-07-26T06:14:30.901696+0000","last_became_active":"2024-07-26T06:14:29.891054+0000","last_became_peered":"2024-07-26T06:14:29.891054+0000","last_unstale":"2024-07-26T06:14:30.901696+0000","last_undegraded":"2024-07-26T06:14:30.901696+0000","last_fullsized":"2024-07-26T06:14:30.901696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:29:25.939484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919321+0000","last_change":"2024-07-26T06:14:30.912787+0000","last_active":"2024-07-26T06:14:31.919321+0000","last_peered":"2024-07-26T06:14:31.919321+0000","last_clean":"2024-07-26T06:14:31.919321+0000","last_became_active":"2024-07-26T06:14:30.912481+0000","last_became_peered":"2024-07-26T06:14:30.912481+0000","last_unstale":"2024-07-26T06:14:31.919321+0000","last_undegraded":"2024-07-26T06:14:31.919321+0000","last_fullsized":"2024-07-26T06:14:31.919321+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:35.760641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942078+0000","last_change":"2024-07-26T06:14:29.896425+0000","last_active":"2024-07-26T06:14:30.942078+0000","last_peered":"2024-07-26T06:14:30.942078+0000","last_clean":"2024-07-26T06:14:30.942078+0000","last_became_active":"2024-07-26T06:14:29.896178+0000","last_became_peered":"2024-07-26T06:14:29.896178+0000","last_unstale":"2024-07-26T06:14:30.942078+0000","last_undegraded":"2024-07-26T06:14:30.942078+0000","last_fullsized":"2024-07-26T06:14:30.942078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:34.749754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919250+0000","last_change":"2024-07-26T06:14:30.914941+0000","last_active":"2024-07-26T06:14:31.919250+0000","last_peered":"2024-07-26T06:14:31.919250+0000","last_clean":"2024-07-26T06:14:31.919250+0000","last_became_active":"2024-07-26T06:14:30.913671+0000","last_became_peered":"2024-07-26T06:14:30.913671+0000","last_unstale":"2024-07-26T06:14:31.919250+0000","last_undegraded":"2024-07-26T06:14:31.919250+0000","last_fullsized":"2024-07-26T06:14:31.919250+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:19.703607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255025+0000","last_change":"2024-07-26T06:14:29.902772+0000","last_active":"2024-07-26T06:14:32.255025+0000","last_peered":"2024-07-26T06:14:32.255025+0000","last_clean":"2024-07-26T06:14:32.255025+0000","last_became_active":"2024-07-26T06:14:29.902290+0000","last_became_peered":"2024-07-26T06:14:29.902290+0000","last_unstale":"2024-07-26T06:14:32.255025+0000","last_undegraded":"2024-07-26T06:14:32.255025+0000","last_fullsized":"2024-07-26T06:14:32.255025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:05.770984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919382+0000","last_change":"2024-07-26T06:14:30.910393+0000","last_active":"2024-07-26T06:14:31.919382+0000","last_peered":"2024-07-26T06:14:31.919382+0000","last_clean":"2024-07-26T06:14:31.919382+0000","last_became_active":"2024-07-26T06:14:30.910073+0000","last_became_peered":"2024-07-26T06:14:30.910073+0000","last_unstale":"2024-07-26T06:14:31.919382+0000","last_undegraded":"2024-07-26T06:14:31.919382+0000","last_fullsized":"2024-07-26T06:14:31.919382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:34:06.163321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.901630+0000","last_change":"2024-07-26T06:14:29.887700+0000","last_active":"2024-07-26T06:14:30.901630+0000","last_peered":"2024-07-26T06:14:30.901630+0000","last_clean":"2024-07-26T06:14:30.901630+0000","last_became_active":"2024-07-26T06:14:29.886992+0000","last_became_peered":"2024-07-26T06:14:29.886992+0000","last_unstale":"2024-07-26T06:14:30.901630+0000","last_undegraded":"2024-07-26T06:14:30.901630+0000","last_fullsized":"2024-07-26T06:14:30.901630+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:24.378070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.945051+0000","last_change":"2024-07-26T06:14:30.938494+0000","last_active":"2024-07-26T06:14:30.945051+0000","last_peered":"2024-07-26T06:14:30.945051+0000","last_clean":"2024-07-26T06:14:30.945051+0000","last_became_active":"2024-07-26T06:14:30.938181+0000","last_became_peered":"2024-07-26T06:14:30.938181+0000","last_unstale":"2024-07-26T06:14:30.945051+0000","last_undegraded":"2024-07-26T06:14:30.945051+0000","last_fullsized":"2024-07-26T06:14:30.945051+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:28:27.267772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.927128+0000","last_change":"2024-07-26T06:14:29.902112+0000","last_active":"2024-07-26T06:14:30.927128+0000","last_peered":"2024-07-26T06:14:30.927128+0000","last_clean":"2024-07-26T06:14:30.927128+0000","last_became_active":"2024-07-26T06:14:29.901767+0000","last_became_peered":"2024-07-26T06:14:29.901767+0000","last_unstale":"2024-07-26T06:14:30.927128+0000","last_undegraded":"2024-07-26T06:14:30.927128+0000","last_fullsized":"2024-07-26T06:14:30.927128+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:15:44.754272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919404+0000","last_change":"2024-07-26T06:14:30.937605+0000","last_active":"2024-07-26T06:14:31.919404+0000","last_peered":"2024-07-26T06:14:31.919404+0000","last_clean":"2024-07-26T06:14:31.919404+0000","last_became_active":"2024-07-26T06:14:30.935167+0000","last_became_peered":"2024-07-26T06:14:30.935167+0000","last_unstale":"2024-07-26T06:14:31.919404+0000","last_undegraded":"2024-07-26T06:14:31.919404+0000","last_fullsized":"2024-07-26T06:14:31.919404+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:39:32.175666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943376+0000","last_change":"2024-07-26T06:14:29.902564+0000","last_active":"2024-07-26T06:14:30.943376+0000","last_peered":"2024-07-26T06:14:30.943376+0000","last_clean":"2024-07-26T06:14:30.943376+0000","last_became_active":"2024-07-26T06:14:29.902265+0000","last_became_peered":"2024-07-26T06:14:29.902265+0000","last_unstale":"2024-07-26T06:14:30.943376+0000","last_undegraded":"2024-07-26T06:14:30.943376+0000","last_fullsized":"2024-07-26T06:14:30.943376+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:06:11.025730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942785+0000","last_change":"2024-07-26T06:14:30.936239+0000","last_active":"2024-07-26T06:14:30.942785+0000","last_peered":"2024-07-26T06:14:30.942785+0000","last_clean":"2024-07-26T06:14:30.942785+0000","last_became_active":"2024-07-26T06:14:30.935650+0000","last_became_peered":"2024-07-26T06:14:30.935650+0000","last_unstale":"2024-07-26T06:14:30.942785+0000","last_undegraded":"2024-07-26T06:14:30.942785+0000","last_fullsized":"2024-07-26T06:14:30.942785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:03.697326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.902468+0000","last_change":"2024-07-26T06:14:29.895723+0000","last_active":"2024-07-26T06:14:30.902468+0000","last_peered":"2024-07-26T06:14:30.902468+0000","last_clean":"2024-07-26T06:14:30.902468+0000","last_became_active":"2024-07-26T06:14:29.895403+0000","last_became_peered":"2024-07-26T06:14:29.895403+0000","last_unstale":"2024-07-26T06:14:30.902468+0000","last_undegraded":"2024-07-26T06:14:30.902468+0000","last_fullsized":"2024-07-26T06:14:30.902468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:18.381616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.931003+0000","last_change":"2024-07-26T06:14:30.923173+0000","last_active":"2024-07-26T06:14:30.931003+0000","last_peered":"2024-07-26T06:14:30.931003+0000","last_clean":"2024-07-26T06:14:30.931003+0000","last_became_active":"2024-07-26T06:14:30.922677+0000","last_became_peered":"2024-07-26T06:14:30.922677+0000","last_unstale":"2024-07-26T06:14:30.931003+0000","last_undegraded":"2024-07-26T06:14:30.931003+0000","last_fullsized":"2024-07-26T06:14:30.931003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:39:32.175666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.917559+0000","last_change":"2024-07-26T06:14:29.890668+0000","last_active":"2024-07-26T06:14:30.917559+0000","last_peered":"2024-07-26T06:14:30.917559+0000","last_clean":"2024-07-26T06:14:30.917559+0000","last_became_active":"2024-07-26T06:14:29.890373+0000","last_became_peered":"2024-07-26T06:14:29.890373+0000","last_unstale":"2024-07-26T06:14:30.917559+0000","last_undegraded":"2024-07-26T06:14:30.917559+0000","last_fullsized":"2024-07-26T06:14:30.917559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:23.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255055+0000","last_change":"2024-07-26T06:14:30.926339+0000","last_active":"2024-07-26T06:14:32.255055+0000","last_peered":"2024-07-26T06:14:32.255055+0000","last_clean":"2024-07-26T06:14:32.255055+0000","last_became_active":"2024-07-26T06:14:30.926006+0000","last_became_peered":"2024-07-26T06:14:30.926006+0000","last_unstale":"2024-07-26T06:14:32.255055+0000","last_undegraded":"2024-07-26T06:14:32.255055+0000","last_fullsized":"2024-07-26T06:14:32.255055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:10:19.889725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.907877+0000","last_change":"2024-07-26T06:14:29.904504+0000","last_active":"2024-07-26T06:14:30.907877+0000","last_peered":"2024-07-26T06:14:30.907877+0000","last_clean":"2024-07-26T06:14:30.907877+0000","last_became_active":"2024-07-26T06:14:29.904005+0000","last_became_peered":"2024-07-26T06:14:29.904005+0000","last_unstale":"2024-07-26T06:14:30.907877+0000","last_undegraded":"2024-07-26T06:14:30.907877+0000","last_fullsized":"2024-07-26T06:14:30.907877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:34:05.152435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940920+0000","last_change":"2024-07-26T06:14:30.931950+0000","last_active":"2024-07-26T06:14:30.940920+0000","last_peered":"2024-07-26T06:14:30.940920+0000","last_clean":"2024-07-26T06:14:30.940920+0000","last_became_active":"2024-07-26T06:14:30.931673+0000","last_became_peered":"2024-07-26T06:14:30.931673+0000","last_unstale":"2024-07-26T06:14:30.940920+0000","last_undegraded":"2024-07-26T06:14:30.940920+0000","last_fullsized":"2024-07-26T06:14:30.940920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:34:47.647661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.913216+0000","last_change":"2024-07-26T06:14:29.901011+0000","last_active":"2024-07-26T06:14:30.913216+0000","last_peered":"2024-07-26T06:14:30.913216+0000","last_clean":"2024-07-26T06:14:30.913216+0000","last_became_active":"2024-07-26T06:14:29.900703+0000","last_became_peered":"2024-07-26T06:14:29.900703+0000","last_unstale":"2024-07-26T06:14:30.913216+0000","last_undegraded":"2024-07-26T06:14:30.913216+0000","last_fullsized":"2024-07-26T06:14:30.913216+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:18.082903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.935069+0000","last_change":"2024-07-26T06:14:30.918227+0000","last_active":"2024-07-26T06:14:30.935069+0000","last_peered":"2024-07-26T06:14:30.935069+0000","last_clean":"2024-07-26T06:14:30.935069+0000","last_became_active":"2024-07-26T06:14:30.917780+0000","last_became_peered":"2024-07-26T06:14:30.917780+0000","last_unstale":"2024-07-26T06:14:30.935069+0000","last_undegraded":"2024-07-26T06:14:30.935069+0000","last_fullsized":"2024-07-26T06:14:30.935069+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:19:59.788774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.902071+0000","last_change":"2024-07-26T06:14:29.904959+0000","last_active":"2024-07-26T06:14:30.902071+0000","last_peered":"2024-07-26T06:14:30.902071+0000","last_clean":"2024-07-26T06:14:30.902071+0000","last_became_active":"2024-07-26T06:14:29.903801+0000","last_became_peered":"2024-07-26T06:14:29.903801+0000","last_unstale":"2024-07-26T06:14:30.902071+0000","last_undegraded":"2024-07-26T06:14:30.902071+0000","last_fullsized":"2024-07-26T06:14:30.902071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:57:17.278062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.929739+0000","last_change":"2024-07-26T06:14:30.917118+0000","last_active":"2024-07-26T06:14:30.929739+0000","last_peered":"2024-07-26T06:14:30.929739+0000","last_clean":"2024-07-26T06:14:30.929739+0000","last_became_active":"2024-07-26T06:14:30.916644+0000","last_became_peered":"2024-07-26T06:14:30.916644+0000","last_unstale":"2024-07-26T06:14:30.929739+0000","last_undegraded":"2024-07-26T06:14:30.929739+0000","last_fullsized":"2024-07-26T06:14:30.929739+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:20:11.701711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.907132+0000","last_change":"2024-07-26T06:14:29.902632+0000","last_active":"2024-07-26T06:14:30.907132+0000","last_peered":"2024-07-26T06:14:30.907132+0000","last_clean":"2024-07-26T06:14:30.907132+0000","last_became_active":"2024-07-26T06:14:29.902365+0000","last_became_peered":"2024-07-26T06:14:29.902365+0000","last_unstale":"2024-07-26T06:14:30.907132+0000","last_undegraded":"2024-07-26T06:14:30.907132+0000","last_fullsized":"2024-07-26T06:14:30.907132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:37:37.293521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939193+0000","last_change":"2024-07-26T06:14:30.931640+0000","last_active":"2024-07-26T06:14:30.939193+0000","last_peered":"2024-07-26T06:14:30.939193+0000","last_clean":"2024-07-26T06:14:30.939193+0000","last_became_active":"2024-07-26T06:14:30.931342+0000","last_became_peered":"2024-07-26T06:14:30.931342+0000","last_unstale":"2024-07-26T06:14:30.939193+0000","last_undegraded":"2024-07-26T06:14:30.939193+0000","last_fullsized":"2024-07-26T06:14:30.939193+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:27:38.310414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255026+0000","last_change":"2024-07-26T06:14:29.909564+0000","last_active":"2024-07-26T06:14:32.255026+0000","last_peered":"2024-07-26T06:14:32.255026+0000","last_clean":"2024-07-26T06:14:32.255026+0000","last_became_active":"2024-07-26T06:14:29.909199+0000","last_became_peered":"2024-07-26T06:14:29.909199+0000","last_unstale":"2024-07-26T06:14:32.255026+0000","last_undegraded":"2024-07-26T06:14:32.255026+0000","last_fullsized":"2024-07-26T06:14:32.255026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:39.524552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":131,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.921971+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:14:30.921971+0000","last_peered":"2024-07-26T06:14:30.921971+0000","last_clean":"2024-07-26T06:14:30.921971+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:14:30.921971+0000","last_undegraded":"2024-07-26T06:14:30.921971+0000","last_fullsized":"2024-07-26T06:14:30.921971+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509457,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":731671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509458,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":141829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":141885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509459,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509458,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":958464,"data_stored":731581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542161,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":141317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":142847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":352256,"data_stored":134438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:14:35.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:14:35.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 -- 172.21.15.163:0/336278364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb720071a70 msgr2=0x7fb7200b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 --2- 172.21.15.163:0/336278364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb720071a70 0x7fb7200b3710 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb72006cf00 tx=0x7fb71802a8b0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 -- 172.21.15.163:0/336278364 shutdown_connections 2024-07-26T06:14:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 --2- 172.21.15.163:0/336278364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb720071a70 0x7fb7200b3710 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 --2- 172.21.15.163:0/336278364 >> v2:172.21.15.164:3300/0 conn(0x7fb7200ce4a0 0x7fb720071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 -- 172.21.15.163:0/336278364 >> 172.21.15.163:0/336278364 conn(0x7fb7200b0e30 msgr2=0x7fb7200b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 -- 172.21.15.163:0/336278364 shutdown_connections 2024-07-26T06:14:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.580+0000 7fb725200640 1 -- 172.21.15.163:0/336278364 wait complete. 2024-07-26T06:14:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.582+0000 7fb725200640 1 Processor -- start 2024-07-26T06:14:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.582+0000 7fb725200640 1 -- start start 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.583+0000 7fb725200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb720071a70 0x7fb7200c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb725200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 0x7fb720159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb725200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7201556f0 0x7fb7201478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb725200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb7200b3fe0 con 0x7fb7200ce4a0 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb725200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb7200b3e60 con 0x7fb720071a70 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb725200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb7200b4160 con 0x7fb7201556f0 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 0x7fb720159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 0x7fb720159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40494/0 (socket says 172.21.15.163:40494) 2024-07-26T06:14:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 -- 172.21.15.163:0/2289707248 learned_addr learned my addr 172.21.15.163:0/2289707248 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb71fe00640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7201556f0 0x7fb7201478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb71f400640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb720071a70 0x7fb7200c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 -- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7201556f0 msgr2=0x7fb7201478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7201556f0 0x7fb7201478c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 -- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb720071a70 msgr2=0x7fb7200c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb720071a70 0x7fb7200c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb717e00640 1 -- 172.21.15.163:0/2289707248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb720158160 con 0x7fb7200ce4a0 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.584+0000 7fb71f400640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb720071a70 0x7fb7200c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.585+0000 7fb717e00640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 0x7fb720159200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb70800d950 tx=0x7fb70800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.585+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708017070 con 0x7fb7200ce4a0 2024-07-26T06:14:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.585+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb708009d20 con 0x7fb7200ce4a0 2024-07-26T06:14:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.585+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb720158440 con 0x7fb7200ce4a0 2024-07-26T06:14:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.585+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7080106c0 con 0x7fb7200ce4a0 2024-07-26T06:14:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.585+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb720158a00 con 0x7fb7200ce4a0 2024-07-26T06:14:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.587+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb720158e40 con 0x7fb7200ce4a0 2024-07-26T06:14:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.588+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb708010860 con 0x7fb7200ce4a0 2024-07-26T06:14:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.588+0000 7fb717400640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6fc0762d0 0x7fb6fc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:35.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.589+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fb708097b90 con 0x7fb7200ce4a0 2024-07-26T06:14:35.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.589+0000 7fb71f400640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6fc0762d0 0x7fb6fc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:35.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.590+0000 7fb71f400640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6fc0762d0 0x7fb6fc078790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb710007fd0 tx=0x7fb710009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.593+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7080614e0 con 0x7fb7200ce4a0 2024-07-26T06:14:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.796+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7200b46f0 con 0x7fb6fc0762d0 2024-07-26T06:14:35.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.808+0000 7fb717400640 1 -- 172.21.15.163:0/2289707248 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251885 (secure 0 0 0) 0x7fb7200b46f0 con 0x7fb6fc0762d0 2024-07-26T06:14:35.814 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:35.815 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:14:35.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6fc0762d0 msgr2=0x7fb6fc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6fc0762d0 0x7fb6fc078790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb710007fd0 tx=0x7fb710009040 comp rx=0 tx=0).stop 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 msgr2=0x7fb720159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 0x7fb720159200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb70800d950 tx=0x7fb70800de10 comp rx=0 tx=0).stop 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 shutdown_connections 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6fc0762d0 0x7fb6fc078790 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7201556f0 0x7fb7201478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7200ce4a0 0x7fb720159200 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 --2- 172.21.15.163:0/2289707248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb720071a70 0x7fb7200c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:35.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 >> 172.21.15.163:0/2289707248 conn(0x7fb7200b0e30 msgr2=0x7fb720151e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:35.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 shutdown_connections 2024-07-26T06:14:35.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:35.815+0000 7fb725200640 1 -- 172.21.15.163:0/2289707248 wait complete. 2024-07-26T06:14:35.835 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-07-26T06:14:35.665063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220252,"kb_used_data":4044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754468,"statfs":{"total":773094113280,"available":772868575232,"internally_reserved":0,"allocated":4141056,"data_stored":2306851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002167"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255593+0000","last_change":"2024-07-26T06:14:30.937043+0000","last_active":"2024-07-26T06:14:32.255593+0000","last_peered":"2024-07-26T06:14:32.255593+0000","last_clean":"2024-07-26T06:14:32.255593+0000","last_became_active":"2024-07-26T06:14:30.936733+0000","last_became_peered":"2024-07-26T06:14:30.936733+0000","last_unstale":"2024-07-26T06:14:32.255593+0000","last_undegraded":"2024-07-26T06:14:32.255593+0000","last_fullsized":"2024-07-26T06:14:32.255593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:16.720050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.930040+0000","last_change":"2024-07-26T06:14:30.914532+0000","last_active":"2024-07-26T06:14:30.930040+0000","last_peered":"2024-07-26T06:14:30.930040+0000","last_clean":"2024-07-26T06:14:30.930040+0000","last_became_active":"2024-07-26T06:14:30.913762+0000","last_became_peered":"2024-07-26T06:14:30.913762+0000","last_unstale":"2024-07-26T06:14:30.930040+0000","last_undegraded":"2024-07-26T06:14:30.930040+0000","last_fullsized":"2024-07-26T06:14:30.930040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:37:40.681640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941895+0000","last_change":"2024-07-26T06:14:30.936293+0000","last_active":"2024-07-26T06:14:30.941895+0000","last_peered":"2024-07-26T06:14:30.941895+0000","last_clean":"2024-07-26T06:14:30.941895+0000","last_became_active":"2024-07-26T06:14:30.935961+0000","last_became_peered":"2024-07-26T06:14:30.935961+0000","last_unstale":"2024-07-26T06:14:30.941895+0000","last_undegraded":"2024-07-26T06:14:30.941895+0000","last_fullsized":"2024-07-26T06:14:30.941895+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:45.290332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939727+0000","last_change":"2024-07-26T06:14:30.935750+0000","last_active":"2024-07-26T06:14:30.939727+0000","last_peered":"2024-07-26T06:14:30.939727+0000","last_clean":"2024-07-26T06:14:30.939727+0000","last_became_active":"2024-07-26T06:14:30.935233+0000","last_became_peered":"2024-07-26T06:14:30.935233+0000","last_unstale":"2024-07-26T06:14:30.939727+0000","last_undegraded":"2024-07-26T06:14:30.939727+0000","last_fullsized":"2024-07-26T06:14:30.939727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:07:02.719589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.918950+0000","last_change":"2024-07-26T06:14:30.909205+0000","last_active":"2024-07-26T06:14:31.918950+0000","last_peered":"2024-07-26T06:14:31.918950+0000","last_clean":"2024-07-26T06:14:31.918950+0000","last_became_active":"2024-07-26T06:14:30.908865+0000","last_became_peered":"2024-07-26T06:14:30.908865+0000","last_unstale":"2024-07-26T06:14:31.918950+0000","last_undegraded":"2024-07-26T06:14:31.918950+0000","last_fullsized":"2024-07-26T06:14:31.918950+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:40.535438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943502+0000","last_change":"2024-07-26T06:14:30.936778+0000","last_active":"2024-07-26T06:14:30.943502+0000","last_peered":"2024-07-26T06:14:30.943502+0000","last_clean":"2024-07-26T06:14:30.943502+0000","last_became_active":"2024-07-26T06:14:30.936465+0000","last_became_peered":"2024-07-26T06:14:30.936465+0000","last_unstale":"2024-07-26T06:14:30.943502+0000","last_undegraded":"2024-07-26T06:14:30.943502+0000","last_fullsized":"2024-07-26T06:14:30.943502+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:27:46.459498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.947216+0000","last_change":"2024-07-26T06:14:30.945157+0000","last_active":"2024-07-26T06:14:30.947216+0000","last_peered":"2024-07-26T06:14:30.947216+0000","last_clean":"2024-07-26T06:14:30.947216+0000","last_became_active":"2024-07-26T06:14:30.944861+0000","last_became_peered":"2024-07-26T06:14:30.944861+0000","last_unstale":"2024-07-26T06:14:30.947216+0000","last_undegraded":"2024-07-26T06:14:30.947216+0000","last_fullsized":"2024-07-26T06:14:30.947216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:46.815340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.936562+0000","last_change":"2024-07-26T06:14:30.928891+0000","last_active":"2024-07-26T06:14:30.936562+0000","last_peered":"2024-07-26T06:14:30.936562+0000","last_clean":"2024-07-26T06:14:30.936562+0000","last_became_active":"2024-07-26T06:14:30.928531+0000","last_became_peered":"2024-07-26T06:14:30.928531+0000","last_unstale":"2024-07-26T06:14:30.936562+0000","last_undegraded":"2024-07-26T06:14:30.936562+0000","last_fullsized":"2024-07-26T06:14:30.936562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:15:45.765158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.938826+0000","last_change":"2024-07-26T06:14:30.922008+0000","last_active":"2024-07-26T06:14:30.938826+0000","last_peered":"2024-07-26T06:14:30.938826+0000","last_clean":"2024-07-26T06:14:30.938826+0000","last_became_active":"2024-07-26T06:14:30.921712+0000","last_became_peered":"2024-07-26T06:14:30.921712+0000","last_unstale":"2024-07-26T06:14:30.938826+0000","last_undegraded":"2024-07-26T06:14:30.938826+0000","last_fullsized":"2024-07-26T06:14:30.938826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:56:58.032909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939310+0000","last_change":"2024-07-26T06:14:30.931244+0000","last_active":"2024-07-26T06:14:30.939310+0000","last_peered":"2024-07-26T06:14:30.939310+0000","last_clean":"2024-07-26T06:14:30.939310+0000","last_became_active":"2024-07-26T06:14:30.930910+0000","last_became_peered":"2024-07-26T06:14:30.930910+0000","last_unstale":"2024-07-26T06:14:30.939310+0000","last_undegraded":"2024-07-26T06:14:30.939310+0000","last_fullsized":"2024-07-26T06:14:30.939310+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:56:58.032909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942546+0000","last_change":"2024-07-26T06:14:30.934242+0000","last_active":"2024-07-26T06:14:30.942546+0000","last_peered":"2024-07-26T06:14:30.942546+0000","last_clean":"2024-07-26T06:14:30.942546+0000","last_became_active":"2024-07-26T06:14:30.933730+0000","last_became_peered":"2024-07-26T06:14:30.933730+0000","last_unstale":"2024-07-26T06:14:30.942546+0000","last_undegraded":"2024-07-26T06:14:30.942546+0000","last_fullsized":"2024-07-26T06:14:30.942546+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:47:58.971707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942101+0000","last_change":"2024-07-26T06:14:30.935656+0000","last_active":"2024-07-26T06:14:30.942101+0000","last_peered":"2024-07-26T06:14:30.942101+0000","last_clean":"2024-07-26T06:14:30.942101+0000","last_became_active":"2024-07-26T06:14:30.935154+0000","last_became_peered":"2024-07-26T06:14:30.935154+0000","last_unstale":"2024-07-26T06:14:30.942101+0000","last_undegraded":"2024-07-26T06:14:30.942101+0000","last_fullsized":"2024-07-26T06:14:30.942101+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:30:47.461579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919091+0000","last_change":"2024-07-26T06:14:30.915521+0000","last_active":"2024-07-26T06:14:31.919091+0000","last_peered":"2024-07-26T06:14:31.919091+0000","last_clean":"2024-07-26T06:14:31.919091+0000","last_became_active":"2024-07-26T06:14:30.914996+0000","last_became_peered":"2024-07-26T06:14:30.914996+0000","last_unstale":"2024-07-26T06:14:31.919091+0000","last_undegraded":"2024-07-26T06:14:31.919091+0000","last_fullsized":"2024-07-26T06:14:31.919091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:19.392502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.932744+0000","last_change":"2024-07-26T06:14:30.917397+0000","last_active":"2024-07-26T06:14:30.932744+0000","last_peered":"2024-07-26T06:14:30.932744+0000","last_clean":"2024-07-26T06:14:30.932744+0000","last_became_active":"2024-07-26T06:14:30.917113+0000","last_became_peered":"2024-07-26T06:14:30.917113+0000","last_unstale":"2024-07-26T06:14:30.932744+0000","last_undegraded":"2024-07-26T06:14:30.932744+0000","last_fullsized":"2024-07-26T06:14:30.932744+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:57.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941278+0000","last_change":"2024-07-26T06:14:30.932610+0000","last_active":"2024-07-26T06:14:30.941278+0000","last_peered":"2024-07-26T06:14:30.941278+0000","last_clean":"2024-07-26T06:14:30.941278+0000","last_became_active":"2024-07-26T06:14:30.931759+0000","last_became_peered":"2024-07-26T06:14:30.931759+0000","last_unstale":"2024-07-26T06:14:30.941278+0000","last_undegraded":"2024-07-26T06:14:30.941278+0000","last_fullsized":"2024-07-26T06:14:30.941278+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:28:27.267772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946859+0000","last_change":"2024-07-26T06:14:30.943069+0000","last_active":"2024-07-26T06:14:30.946859+0000","last_peered":"2024-07-26T06:14:30.946859+0000","last_clean":"2024-07-26T06:14:30.946859+0000","last_became_active":"2024-07-26T06:14:30.942190+0000","last_became_peered":"2024-07-26T06:14:30.942190+0000","last_unstale":"2024-07-26T06:14:30.946859+0000","last_undegraded":"2024-07-26T06:14:30.946859+0000","last_fullsized":"2024-07-26T06:14:30.946859+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:45.290332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944656+0000","last_change":"2024-07-26T06:14:30.938725+0000","last_active":"2024-07-26T06:14:30.944656+0000","last_peered":"2024-07-26T06:14:30.944656+0000","last_clean":"2024-07-26T06:14:30.944656+0000","last_became_active":"2024-07-26T06:14:30.938420+0000","last_became_peered":"2024-07-26T06:14:30.938420+0000","last_unstale":"2024-07-26T06:14:30.944656+0000","last_undegraded":"2024-07-26T06:14:30.944656+0000","last_fullsized":"2024-07-26T06:14:30.944656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:16.720050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.932783+0000","last_change":"2024-07-26T06:14:30.918857+0000","last_active":"2024-07-26T06:14:30.932783+0000","last_peered":"2024-07-26T06:14:30.932783+0000","last_clean":"2024-07-26T06:14:30.932783+0000","last_became_active":"2024-07-26T06:14:30.918601+0000","last_became_peered":"2024-07-26T06:14:30.918601+0000","last_unstale":"2024-07-26T06:14:30.932783+0000","last_undegraded":"2024-07-26T06:14:30.932783+0000","last_fullsized":"2024-07-26T06:14:30.932783+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:45:03.127108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.933342+0000","last_change":"2024-07-26T06:14:30.923161+0000","last_active":"2024-07-26T06:14:30.933342+0000","last_peered":"2024-07-26T06:14:30.933342+0000","last_clean":"2024-07-26T06:14:30.933342+0000","last_became_active":"2024-07-26T06:14:30.922893+0000","last_became_peered":"2024-07-26T06:14:30.922893+0000","last_unstale":"2024-07-26T06:14:30.933342+0000","last_undegraded":"2024-07-26T06:14:30.933342+0000","last_fullsized":"2024-07-26T06:14:30.933342+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:12:49.364457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919057+0000","last_change":"2024-07-26T06:14:30.939350+0000","last_active":"2024-07-26T06:14:31.919057+0000","last_peered":"2024-07-26T06:14:31.919057+0000","last_clean":"2024-07-26T06:14:31.919057+0000","last_became_active":"2024-07-26T06:14:30.938968+0000","last_became_peered":"2024-07-26T06:14:30.938968+0000","last_unstale":"2024-07-26T06:14:31.919057+0000","last_undegraded":"2024-07-26T06:14:31.919057+0000","last_fullsized":"2024-07-26T06:14:31.919057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:06.781871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.930216+0000","last_change":"2024-07-26T06:14:30.919374+0000","last_active":"2024-07-26T06:14:30.930216+0000","last_peered":"2024-07-26T06:14:30.930216+0000","last_clean":"2024-07-26T06:14:30.930216+0000","last_became_active":"2024-07-26T06:14:30.918846+0000","last_became_peered":"2024-07-26T06:14:30.918846+0000","last_unstale":"2024-07-26T06:14:30.930216+0000","last_undegraded":"2024-07-26T06:14:30.930216+0000","last_fullsized":"2024-07-26T06:14:30.930216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:48:13.625747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940323+0000","last_change":"2024-07-26T06:14:30.932146+0000","last_active":"2024-07-26T06:14:30.940323+0000","last_peered":"2024-07-26T06:14:30.940323+0000","last_clean":"2024-07-26T06:14:30.940323+0000","last_became_active":"2024-07-26T06:14:30.931859+0000","last_became_peered":"2024-07-26T06:14:30.931859+0000","last_unstale":"2024-07-26T06:14:30.940323+0000","last_undegraded":"2024-07-26T06:14:30.940323+0000","last_fullsized":"2024-07-26T06:14:30.940323+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:21:47.556217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.937561+0000","last_change":"2024-07-26T06:14:30.929342+0000","last_active":"2024-07-26T06:14:30.937561+0000","last_peered":"2024-07-26T06:14:30.937561+0000","last_clean":"2024-07-26T06:14:30.937561+0000","last_became_active":"2024-07-26T06:14:30.929046+0000","last_became_peered":"2024-07-26T06:14:30.929046+0000","last_unstale":"2024-07-26T06:14:30.937561+0000","last_undegraded":"2024-07-26T06:14:30.937561+0000","last_fullsized":"2024-07-26T06:14:30.937561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:54.664264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941035+0000","last_change":"2024-07-26T06:14:30.928572+0000","last_active":"2024-07-26T06:14:30.941035+0000","last_peered":"2024-07-26T06:14:30.941035+0000","last_clean":"2024-07-26T06:14:30.941035+0000","last_became_active":"2024-07-26T06:14:30.928246+0000","last_became_peered":"2024-07-26T06:14:30.928246+0000","last_unstale":"2024-07-26T06:14:30.941035+0000","last_undegraded":"2024-07-26T06:14:30.941035+0000","last_fullsized":"2024-07-26T06:14:30.941035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:10:19.889725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939643+0000","last_change":"2024-07-26T06:14:30.932616+0000","last_active":"2024-07-26T06:14:30.939643+0000","last_peered":"2024-07-26T06:14:30.939643+0000","last_clean":"2024-07-26T06:14:30.939643+0000","last_became_active":"2024-07-26T06:14:30.932352+0000","last_became_peered":"2024-07-26T06:14:30.932352+0000","last_unstale":"2024-07-26T06:14:30.939643+0000","last_undegraded":"2024-07-26T06:14:30.939643+0000","last_fullsized":"2024-07-26T06:14:30.939643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:09.397816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.934120+0000","last_change":"2024-07-26T06:14:30.930180+0000","last_active":"2024-07-26T06:14:30.934120+0000","last_peered":"2024-07-26T06:14:30.934120+0000","last_clean":"2024-07-26T06:14:30.934120+0000","last_became_active":"2024-07-26T06:14:30.929873+0000","last_became_peered":"2024-07-26T06:14:30.929873+0000","last_unstale":"2024-07-26T06:14:30.934120+0000","last_undegraded":"2024-07-26T06:14:30.934120+0000","last_fullsized":"2024-07-26T06:14:30.934120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:22.448217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939828+0000","last_change":"2024-07-26T06:14:30.936704+0000","last_active":"2024-07-26T06:14:30.939828+0000","last_peered":"2024-07-26T06:14:30.939828+0000","last_clean":"2024-07-26T06:14:30.939828+0000","last_became_active":"2024-07-26T06:14:30.936074+0000","last_became_peered":"2024-07-26T06:14:30.936074+0000","last_unstale":"2024-07-26T06:14:30.939828+0000","last_undegraded":"2024-07-26T06:14:30.939828+0000","last_fullsized":"2024-07-26T06:14:30.939828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:37:38.304407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940786+0000","last_change":"2024-07-26T06:14:30.930555+0000","last_active":"2024-07-26T06:14:30.940786+0000","last_peered":"2024-07-26T06:14:30.940786+0000","last_clean":"2024-07-26T06:14:30.940786+0000","last_became_active":"2024-07-26T06:14:30.930226+0000","last_became_peered":"2024-07-26T06:14:30.930226+0000","last_unstale":"2024-07-26T06:14:30.940786+0000","last_undegraded":"2024-07-26T06:14:30.940786+0000","last_fullsized":"2024-07-26T06:14:30.940786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:27:38.310414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.929573+0000","last_change":"2024-07-26T06:14:30.918965+0000","last_active":"2024-07-26T06:14:30.929573+0000","last_peered":"2024-07-26T06:14:30.929573+0000","last_clean":"2024-07-26T06:14:30.929573+0000","last_became_active":"2024-07-26T06:14:30.918648+0000","last_became_peered":"2024-07-26T06:14:30.918648+0000","last_unstale":"2024-07-26T06:14:30.929573+0000","last_undegraded":"2024-07-26T06:14:30.929573+0000","last_fullsized":"2024-07-26T06:14:30.929573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:24.078491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942168+0000","last_change":"2024-07-26T06:14:30.929862+0000","last_active":"2024-07-26T06:14:30.942168+0000","last_peered":"2024-07-26T06:14:30.942168+0000","last_clean":"2024-07-26T06:14:30.942168+0000","last_became_active":"2024-07-26T06:14:30.929588+0000","last_became_peered":"2024-07-26T06:14:30.929588+0000","last_unstale":"2024-07-26T06:14:30.942168+0000","last_undegraded":"2024-07-26T06:14:30.942168+0000","last_fullsized":"2024-07-26T06:14:30.942168+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:10:19.889725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255203+0000","last_change":"2024-07-26T06:14:30.922576+0000","last_active":"2024-07-26T06:14:32.255203+0000","last_peered":"2024-07-26T06:14:32.255203+0000","last_clean":"2024-07-26T06:14:32.255203+0000","last_became_active":"2024-07-26T06:14:30.922225+0000","last_became_peered":"2024-07-26T06:14:30.922225+0000","last_unstale":"2024-07-26T06:14:32.255203+0000","last_undegraded":"2024-07-26T06:14:32.255203+0000","last_fullsized":"2024-07-26T06:14:32.255203+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:27:38.310414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946840+0000","last_change":"2024-07-26T06:14:30.942849+0000","last_active":"2024-07-26T06:14:30.946840+0000","last_peered":"2024-07-26T06:14:30.946840+0000","last_clean":"2024-07-26T06:14:30.946840+0000","last_became_active":"2024-07-26T06:14:30.941822+0000","last_became_peered":"2024-07-26T06:14:30.941822+0000","last_unstale":"2024-07-26T06:14:30.946840+0000","last_undegraded":"2024-07-26T06:14:30.946840+0000","last_fullsized":"2024-07-26T06:14:30.946840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:03.697326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941404+0000","last_change":"2024-07-26T06:14:30.936955+0000","last_active":"2024-07-26T06:14:30.941404+0000","last_peered":"2024-07-26T06:14:30.941404+0000","last_clean":"2024-07-26T06:14:30.941404+0000","last_became_active":"2024-07-26T06:14:30.936554+0000","last_became_peered":"2024-07-26T06:14:30.936554+0000","last_unstale":"2024-07-26T06:14:30.941404+0000","last_undegraded":"2024-07-26T06:14:30.941404+0000","last_fullsized":"2024-07-26T06:14:30.941404+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:15:45.765158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255694+0000","last_change":"2024-07-26T06:14:29.904778+0000","last_active":"2024-07-26T06:14:32.255694+0000","last_peered":"2024-07-26T06:14:32.255694+0000","last_clean":"2024-07-26T06:14:32.255694+0000","last_became_active":"2024-07-26T06:14:29.904495+0000","last_became_peered":"2024-07-26T06:14:29.904495+0000","last_unstale":"2024-07-26T06:14:32.255694+0000","last_undegraded":"2024-07-26T06:14:32.255694+0000","last_fullsized":"2024-07-26T06:14:32.255694+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:18.381616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255538+0000","last_change":"2024-07-26T06:14:30.940539+0000","last_active":"2024-07-26T06:14:32.255538+0000","last_peered":"2024-07-26T06:14:32.255538+0000","last_clean":"2024-07-26T06:14:32.255538+0000","last_became_active":"2024-07-26T06:14:30.939690+0000","last_became_peered":"2024-07-26T06:14:30.939690+0000","last_unstale":"2024-07-26T06:14:32.255538+0000","last_undegraded":"2024-07-26T06:14:32.255538+0000","last_fullsized":"2024-07-26T06:14:32.255538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:28:27.267772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.936797+0000","last_change":"2024-07-26T06:14:29.902273+0000","last_active":"2024-07-26T06:14:30.936797+0000","last_peered":"2024-07-26T06:14:30.936797+0000","last_clean":"2024-07-26T06:14:30.936797+0000","last_became_active":"2024-07-26T06:14:29.901774+0000","last_became_peered":"2024-07-26T06:14:29.901774+0000","last_unstale":"2024-07-26T06:14:30.936797+0000","last_undegraded":"2024-07-26T06:14:30.936797+0000","last_fullsized":"2024-07-26T06:14:30.936797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:05.770984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.931569+0000","last_change":"2024-07-26T06:14:30.918223+0000","last_active":"2024-07-26T06:14:30.931569+0000","last_peered":"2024-07-26T06:14:30.931569+0000","last_clean":"2024-07-26T06:14:30.931569+0000","last_became_active":"2024-07-26T06:14:30.917834+0000","last_became_peered":"2024-07-26T06:14:30.917834+0000","last_unstale":"2024-07-26T06:14:30.931569+0000","last_undegraded":"2024-07-26T06:14:30.931569+0000","last_fullsized":"2024-07-26T06:14:30.931569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:06:12.036616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.934190+0000","last_change":"2024-07-26T06:14:29.887831+0000","last_active":"2024-07-26T06:14:30.934190+0000","last_peered":"2024-07-26T06:14:30.934190+0000","last_clean":"2024-07-26T06:14:30.934190+0000","last_became_active":"2024-07-26T06:14:29.887096+0000","last_became_peered":"2024-07-26T06:14:29.887096+0000","last_unstale":"2024-07-26T06:14:30.934190+0000","last_undegraded":"2024-07-26T06:14:30.934190+0000","last_fullsized":"2024-07-26T06:14:30.934190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:48:58.632151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939808+0000","last_change":"2024-07-26T06:14:30.936498+0000","last_active":"2024-07-26T06:14:30.939808+0000","last_peered":"2024-07-26T06:14:30.939808+0000","last_clean":"2024-07-26T06:14:30.939808+0000","last_became_active":"2024-07-26T06:14:30.935664+0000","last_became_peered":"2024-07-26T06:14:30.935664+0000","last_unstale":"2024-07-26T06:14:30.939808+0000","last_undegraded":"2024-07-26T06:14:30.939808+0000","last_fullsized":"2024-07-26T06:14:30.939808+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:56:02.763778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919170+0000","last_change":"2024-07-26T06:14:29.892846+0000","last_active":"2024-07-26T06:14:31.919170+0000","last_peered":"2024-07-26T06:14:31.919170+0000","last_clean":"2024-07-26T06:14:31.919170+0000","last_became_active":"2024-07-26T06:14:29.890684+0000","last_became_peered":"2024-07-26T06:14:29.890684+0000","last_unstale":"2024-07-26T06:14:31.919170+0000","last_undegraded":"2024-07-26T06:14:31.919170+0000","last_fullsized":"2024-07-26T06:14:31.919170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:44:55.950964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944544+0000","last_change":"2024-07-26T06:14:30.939827+0000","last_active":"2024-07-26T06:14:30.944544+0000","last_peered":"2024-07-26T06:14:30.944544+0000","last_clean":"2024-07-26T06:14:30.944544+0000","last_became_active":"2024-07-26T06:14:30.939558+0000","last_became_peered":"2024-07-26T06:14:30.939558+0000","last_unstale":"2024-07-26T06:14:30.944544+0000","last_undegraded":"2024-07-26T06:14:30.944544+0000","last_fullsized":"2024-07-26T06:14:30.944544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:30:47.461579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941682+0000","last_change":"2024-07-26T06:14:29.886269+0000","last_active":"2024-07-26T06:14:30.941682+0000","last_peered":"2024-07-26T06:14:30.941682+0000","last_clean":"2024-07-26T06:14:30.941682+0000","last_became_active":"2024-07-26T06:14:29.885560+0000","last_became_peered":"2024-07-26T06:14:29.885560+0000","last_unstale":"2024-07-26T06:14:30.941682+0000","last_undegraded":"2024-07-26T06:14:30.941682+0000","last_fullsized":"2024-07-26T06:14:30.941682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:27:21.475143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944177+0000","last_change":"2024-07-26T06:14:30.938191+0000","last_active":"2024-07-26T06:14:30.944177+0000","last_peered":"2024-07-26T06:14:30.944177+0000","last_clean":"2024-07-26T06:14:30.944177+0000","last_became_active":"2024-07-26T06:14:30.937813+0000","last_became_peered":"2024-07-26T06:14:30.937813+0000","last_unstale":"2024-07-26T06:14:30.944177+0000","last_undegraded":"2024-07-26T06:14:30.944177+0000","last_fullsized":"2024-07-26T06:14:30.944177+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:09:46.815340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940310+0000","last_change":"2024-07-26T06:14:29.889694+0000","last_active":"2024-07-26T06:14:30.940310+0000","last_peered":"2024-07-26T06:14:30.940310+0000","last_clean":"2024-07-26T06:14:30.940310+0000","last_became_active":"2024-07-26T06:14:29.889029+0000","last_became_peered":"2024-07-26T06:14:29.889029+0000","last_unstale":"2024-07-26T06:14:30.940310+0000","last_undegraded":"2024-07-26T06:14:30.940310+0000","last_fullsized":"2024-07-26T06:14:30.940310+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:28:10.263397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.938025+0000","last_change":"2024-07-26T06:14:30.925886+0000","last_active":"2024-07-26T06:14:30.938025+0000","last_peered":"2024-07-26T06:14:30.938025+0000","last_clean":"2024-07-26T06:14:30.938025+0000","last_became_active":"2024-07-26T06:14:30.925583+0000","last_became_peered":"2024-07-26T06:14:30.925583+0000","last_unstale":"2024-07-26T06:14:30.938025+0000","last_undegraded":"2024-07-26T06:14:30.938025+0000","last_fullsized":"2024-07-26T06:14:30.938025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:06:09.397816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.906920+0000","last_change":"2024-07-26T06:14:29.891691+0000","last_active":"2024-07-26T06:14:30.906920+0000","last_peered":"2024-07-26T06:14:30.906920+0000","last_clean":"2024-07-26T06:14:30.906920+0000","last_became_active":"2024-07-26T06:14:29.891434+0000","last_became_peered":"2024-07-26T06:14:29.891434+0000","last_unstale":"2024-07-26T06:14:30.906920+0000","last_undegraded":"2024-07-26T06:14:30.906920+0000","last_fullsized":"2024-07-26T06:14:30.906920+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:56:01.752892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939615+0000","last_change":"2024-07-26T06:14:30.934878+0000","last_active":"2024-07-26T06:14:30.939615+0000","last_peered":"2024-07-26T06:14:30.939615+0000","last_clean":"2024-07-26T06:14:30.939615+0000","last_became_active":"2024-07-26T06:14:30.934577+0000","last_became_peered":"2024-07-26T06:14:30.934577+0000","last_unstale":"2024-07-26T06:14:30.939615+0000","last_undegraded":"2024-07-26T06:14:30.939615+0000","last_fullsized":"2024-07-26T06:14:30.939615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:21:47.556217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.905926+0000","last_change":"2024-07-26T06:14:29.899602+0000","last_active":"2024-07-26T06:14:30.905926+0000","last_peered":"2024-07-26T06:14:30.905926+0000","last_clean":"2024-07-26T06:14:30.905926+0000","last_became_active":"2024-07-26T06:14:29.899306+0000","last_became_peered":"2024-07-26T06:14:29.899306+0000","last_unstale":"2024-07-26T06:14:30.905926+0000","last_undegraded":"2024-07-26T06:14:30.905926+0000","last_fullsized":"2024-07-26T06:14:30.905926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:53.699963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941803+0000","last_change":"2024-07-26T06:14:30.934708+0000","last_active":"2024-07-26T06:14:30.941803+0000","last_peered":"2024-07-26T06:14:30.941803+0000","last_clean":"2024-07-26T06:14:30.941803+0000","last_became_active":"2024-07-26T06:14:30.934421+0000","last_became_peered":"2024-07-26T06:14:30.934421+0000","last_unstale":"2024-07-26T06:14:30.941803+0000","last_undegraded":"2024-07-26T06:14:30.941803+0000","last_fullsized":"2024-07-26T06:14:30.941803+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:16.720050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.910525+0000","last_change":"2024-07-26T06:14:29.902854+0000","last_active":"2024-07-26T06:14:30.910525+0000","last_peered":"2024-07-26T06:14:30.910525+0000","last_clean":"2024-07-26T06:14:30.910525+0000","last_became_active":"2024-07-26T06:14:29.901454+0000","last_became_peered":"2024-07-26T06:14:29.901454+0000","last_unstale":"2024-07-26T06:14:30.910525+0000","last_undegraded":"2024-07-26T06:14:30.910525+0000","last_fullsized":"2024-07-26T06:14:30.910525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:30.425545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.932701+0000","last_change":"2024-07-26T06:14:30.917577+0000","last_active":"2024-07-26T06:14:30.932701+0000","last_peered":"2024-07-26T06:14:30.932701+0000","last_clean":"2024-07-26T06:14:30.932701+0000","last_became_active":"2024-07-26T06:14:30.917214+0000","last_became_peered":"2024-07-26T06:14:30.917214+0000","last_unstale":"2024-07-26T06:14:30.932701+0000","last_undegraded":"2024-07-26T06:14:30.932701+0000","last_fullsized":"2024-07-26T06:14:30.932701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:53.787034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.903784+0000","last_change":"2024-07-26T06:14:29.899851+0000","last_active":"2024-07-26T06:14:30.903784+0000","last_peered":"2024-07-26T06:14:30.903784+0000","last_clean":"2024-07-26T06:14:30.903784+0000","last_became_active":"2024-07-26T06:14:29.899536+0000","last_became_peered":"2024-07-26T06:14:29.899536+0000","last_unstale":"2024-07-26T06:14:30.903784+0000","last_undegraded":"2024-07-26T06:14:30.903784+0000","last_fullsized":"2024-07-26T06:14:30.903784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:18.692721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939004+0000","last_change":"2024-07-26T06:14:30.934383+0000","last_active":"2024-07-26T06:14:30.939004+0000","last_peered":"2024-07-26T06:14:30.939004+0000","last_clean":"2024-07-26T06:14:30.939004+0000","last_became_active":"2024-07-26T06:14:30.933971+0000","last_became_peered":"2024-07-26T06:14:30.933971+0000","last_unstale":"2024-07-26T06:14:30.939004+0000","last_undegraded":"2024-07-26T06:14:30.939004+0000","last_fullsized":"2024-07-26T06:14:30.939004+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:30:49.519398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943265+0000","last_change":"2024-07-26T06:14:29.902544+0000","last_active":"2024-07-26T06:14:30.943265+0000","last_peered":"2024-07-26T06:14:30.943265+0000","last_clean":"2024-07-26T06:14:30.943265+0000","last_became_active":"2024-07-26T06:14:29.902256+0000","last_became_peered":"2024-07-26T06:14:29.902256+0000","last_unstale":"2024-07-26T06:14:30.943265+0000","last_undegraded":"2024-07-26T06:14:30.943265+0000","last_fullsized":"2024-07-26T06:14:30.943265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:47:57.960821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943919+0000","last_change":"2024-07-26T06:14:30.936996+0000","last_active":"2024-07-26T06:14:30.943919+0000","last_peered":"2024-07-26T06:14:30.943919+0000","last_clean":"2024-07-26T06:14:30.943919+0000","last_became_active":"2024-07-26T06:14:30.935930+0000","last_became_peered":"2024-07-26T06:14:30.935930+0000","last_unstale":"2024-07-26T06:14:30.943919+0000","last_undegraded":"2024-07-26T06:14:30.943919+0000","last_fullsized":"2024-07-26T06:14:30.943919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:03:33.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946561+0000","last_change":"2024-07-26T06:14:29.890840+0000","last_active":"2024-07-26T06:14:30.946561+0000","last_peered":"2024-07-26T06:14:30.946561+0000","last_clean":"2024-07-26T06:14:30.946561+0000","last_became_active":"2024-07-26T06:14:29.886784+0000","last_became_peered":"2024-07-26T06:14:29.886784+0000","last_unstale":"2024-07-26T06:14:30.946561+0000","last_undegraded":"2024-07-26T06:14:30.946561+0000","last_fullsized":"2024-07-26T06:14:30.946561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:32:05.124118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939681+0000","last_change":"2024-07-26T06:14:30.935028+0000","last_active":"2024-07-26T06:14:30.939681+0000","last_peered":"2024-07-26T06:14:30.939681+0000","last_clean":"2024-07-26T06:14:30.939681+0000","last_became_active":"2024-07-26T06:14:30.934647+0000","last_became_peered":"2024-07-26T06:14:30.934647+0000","last_unstale":"2024-07-26T06:14:30.939681+0000","last_undegraded":"2024-07-26T06:14:30.939681+0000","last_fullsized":"2024-07-26T06:14:30.939681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:37:24.270350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944842+0000","last_change":"2024-07-26T06:14:29.899714+0000","last_active":"2024-07-26T06:14:30.944842+0000","last_peered":"2024-07-26T06:14:30.944842+0000","last_clean":"2024-07-26T06:14:30.944842+0000","last_became_active":"2024-07-26T06:14:29.899402+0000","last_became_peered":"2024-07-26T06:14:29.899402+0000","last_unstale":"2024-07-26T06:14:30.944842+0000","last_undegraded":"2024-07-26T06:14:30.944842+0000","last_fullsized":"2024-07-26T06:14:30.944842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:34:06.610898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.933246+0000","last_change":"2024-07-26T06:14:30.919761+0000","last_active":"2024-07-26T06:14:30.933246+0000","last_peered":"2024-07-26T06:14:30.933246+0000","last_clean":"2024-07-26T06:14:30.933246+0000","last_became_active":"2024-07-26T06:14:30.919453+0000","last_became_peered":"2024-07-26T06:14:30.919453+0000","last_unstale":"2024-07-26T06:14:30.933246+0000","last_undegraded":"2024-07-26T06:14:30.933246+0000","last_fullsized":"2024-07-26T06:14:30.933246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:36:53.985771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.913074+0000","last_change":"2024-07-26T06:14:29.894224+0000","last_active":"2024-07-26T06:14:30.913074+0000","last_peered":"2024-07-26T06:14:30.913074+0000","last_clean":"2024-07-26T06:14:30.913074+0000","last_became_active":"2024-07-26T06:14:29.893561+0000","last_became_peered":"2024-07-26T06:14:29.893561+0000","last_unstale":"2024-07-26T06:14:30.913074+0000","last_undegraded":"2024-07-26T06:14:30.913074+0000","last_fullsized":"2024-07-26T06:14:30.913074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:53.699963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946874+0000","last_change":"2024-07-26T06:14:30.943291+0000","last_active":"2024-07-26T06:14:30.946874+0000","last_peered":"2024-07-26T06:14:30.946874+0000","last_clean":"2024-07-26T06:14:30.946874+0000","last_became_active":"2024-07-26T06:14:30.942541+0000","last_became_peered":"2024-07-26T06:14:30.942541+0000","last_unstale":"2024-07-26T06:14:30.946874+0000","last_undegraded":"2024-07-26T06:14:30.946874+0000","last_fullsized":"2024-07-26T06:14:30.946874+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:03:33.301254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.906732+0000","last_change":"2024-07-26T06:14:29.900504+0000","last_active":"2024-07-26T06:14:30.906732+0000","last_peered":"2024-07-26T06:14:30.906732+0000","last_clean":"2024-07-26T06:14:30.906732+0000","last_became_active":"2024-07-26T06:14:29.900189+0000","last_became_peered":"2024-07-26T06:14:29.900189+0000","last_unstale":"2024-07-26T06:14:30.906732+0000","last_undegraded":"2024-07-26T06:14:30.906732+0000","last_fullsized":"2024-07-26T06:14:30.906732+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:07:01.708703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255421+0000","last_change":"2024-07-26T06:14:30.932244+0000","last_active":"2024-07-26T06:14:32.255421+0000","last_peered":"2024-07-26T06:14:32.255421+0000","last_clean":"2024-07-26T06:14:32.255421+0000","last_became_active":"2024-07-26T06:14:30.930521+0000","last_became_peered":"2024-07-26T06:14:30.930521+0000","last_unstale":"2024-07-26T06:14:32.255421+0000","last_undegraded":"2024-07-26T06:14:32.255421+0000","last_fullsized":"2024-07-26T06:14:32.255421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:47:58.971707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.946696+0000","last_change":"2024-07-26T06:14:29.883552+0000","last_active":"2024-07-26T06:14:30.946696+0000","last_peered":"2024-07-26T06:14:30.946696+0000","last_clean":"2024-07-26T06:14:30.946696+0000","last_became_active":"2024-07-26T06:14:29.883251+0000","last_became_peered":"2024-07-26T06:14:29.883251+0000","last_unstale":"2024-07-26T06:14:30.946696+0000","last_undegraded":"2024-07-26T06:14:30.946696+0000","last_fullsized":"2024-07-26T06:14:30.946696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:32.530603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.941662+0000","last_change":"2024-07-26T06:14:30.935702+0000","last_active":"2024-07-26T06:14:30.941662+0000","last_peered":"2024-07-26T06:14:30.941662+0000","last_clean":"2024-07-26T06:14:30.941662+0000","last_became_active":"2024-07-26T06:14:30.935398+0000","last_became_peered":"2024-07-26T06:14:30.935398+0000","last_unstale":"2024-07-26T06:14:30.941662+0000","last_undegraded":"2024-07-26T06:14:30.941662+0000","last_fullsized":"2024-07-26T06:14:30.941662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:37:24.270350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.936061+0000","last_change":"2024-07-26T06:14:29.884910+0000","last_active":"2024-07-26T06:14:30.936061+0000","last_peered":"2024-07-26T06:14:30.936061+0000","last_clean":"2024-07-26T06:14:30.936061+0000","last_became_active":"2024-07-26T06:14:29.883104+0000","last_became_peered":"2024-07-26T06:14:29.883104+0000","last_unstale":"2024-07-26T06:14:30.936061+0000","last_undegraded":"2024-07-26T06:14:30.936061+0000","last_fullsized":"2024-07-26T06:14:30.936061+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:20:10.690825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.945316+0000","last_change":"2024-07-26T06:14:30.942257+0000","last_active":"2024-07-26T06:14:30.945316+0000","last_peered":"2024-07-26T06:14:30.945316+0000","last_clean":"2024-07-26T06:14:30.945316+0000","last_became_active":"2024-07-26T06:14:30.941542+0000","last_became_peered":"2024-07-26T06:14:30.941542+0000","last_unstale":"2024-07-26T06:14:30.945316+0000","last_undegraded":"2024-07-26T06:14:30.945316+0000","last_fullsized":"2024-07-26T06:14:30.945316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:15.365238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.944451+0000","last_change":"2024-07-26T06:14:29.898758+0000","last_active":"2024-07-26T06:14:30.944451+0000","last_peered":"2024-07-26T06:14:30.944451+0000","last_clean":"2024-07-26T06:14:30.944451+0000","last_became_active":"2024-07-26T06:14:29.898498+0000","last_became_peered":"2024-07-26T06:14:29.898498+0000","last_unstale":"2024-07-26T06:14:30.944451+0000","last_undegraded":"2024-07-26T06:14:30.944451+0000","last_fullsized":"2024-07-26T06:14:30.944451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:18.082903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.922014+0000","last_change":"2024-07-26T06:14:30.918622+0000","last_active":"2024-07-26T06:14:30.922014+0000","last_peered":"2024-07-26T06:14:30.922014+0000","last_clean":"2024-07-26T06:14:30.922014+0000","last_became_active":"2024-07-26T06:14:30.918322+0000","last_became_peered":"2024-07-26T06:14:30.918322+0000","last_unstale":"2024-07-26T06:14:30.922014+0000","last_undegraded":"2024-07-26T06:14:30.922014+0000","last_fullsized":"2024-07-26T06:14:30.922014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:48:13.625747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939703+0000","last_change":"2024-07-26T06:14:29.889911+0000","last_active":"2024-07-26T06:14:30.939703+0000","last_peered":"2024-07-26T06:14:30.939703+0000","last_clean":"2024-07-26T06:14:30.939703+0000","last_became_active":"2024-07-26T06:14:29.889021+0000","last_became_peered":"2024-07-26T06:14:29.889021+0000","last_unstale":"2024-07-26T06:14:30.939703+0000","last_undegraded":"2024-07-26T06:14:30.939703+0000","last_fullsized":"2024-07-26T06:14:30.939703+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:48.165552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919716+0000","last_change":"2024-07-26T06:14:30.931601+0000","last_active":"2024-07-26T06:14:31.919716+0000","last_peered":"2024-07-26T06:14:31.919716+0000","last_clean":"2024-07-26T06:14:31.919716+0000","last_became_active":"2024-07-26T06:14:30.930659+0000","last_became_peered":"2024-07-26T06:14:30.930659+0000","last_unstale":"2024-07-26T06:14:31.919716+0000","last_undegraded":"2024-07-26T06:14:31.919716+0000","last_fullsized":"2024-07-26T06:14:31.919716+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:54.664264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.901696+0000","last_change":"2024-07-26T06:14:29.891888+0000","last_active":"2024-07-26T06:14:30.901696+0000","last_peered":"2024-07-26T06:14:30.901696+0000","last_clean":"2024-07-26T06:14:30.901696+0000","last_became_active":"2024-07-26T06:14:29.891054+0000","last_became_peered":"2024-07-26T06:14:29.891054+0000","last_unstale":"2024-07-26T06:14:30.901696+0000","last_undegraded":"2024-07-26T06:14:30.901696+0000","last_fullsized":"2024-07-26T06:14:30.901696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:29:25.939484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919321+0000","last_change":"2024-07-26T06:14:30.912787+0000","last_active":"2024-07-26T06:14:31.919321+0000","last_peered":"2024-07-26T06:14:31.919321+0000","last_clean":"2024-07-26T06:14:31.919321+0000","last_became_active":"2024-07-26T06:14:30.912481+0000","last_became_peered":"2024-07-26T06:14:30.912481+0000","last_unstale":"2024-07-26T06:14:31.919321+0000","last_undegraded":"2024-07-26T06:14:31.919321+0000","last_fullsized":"2024-07-26T06:14:31.919321+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:35.760641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942078+0000","last_change":"2024-07-26T06:14:29.896425+0000","last_active":"2024-07-26T06:14:30.942078+0000","last_peered":"2024-07-26T06:14:30.942078+0000","last_clean":"2024-07-26T06:14:30.942078+0000","last_became_active":"2024-07-26T06:14:29.896178+0000","last_became_peered":"2024-07-26T06:14:29.896178+0000","last_unstale":"2024-07-26T06:14:30.942078+0000","last_undegraded":"2024-07-26T06:14:30.942078+0000","last_fullsized":"2024-07-26T06:14:30.942078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:34.749754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919250+0000","last_change":"2024-07-26T06:14:30.914941+0000","last_active":"2024-07-26T06:14:31.919250+0000","last_peered":"2024-07-26T06:14:31.919250+0000","last_clean":"2024-07-26T06:14:31.919250+0000","last_became_active":"2024-07-26T06:14:30.913671+0000","last_became_peered":"2024-07-26T06:14:30.913671+0000","last_unstale":"2024-07-26T06:14:31.919250+0000","last_undegraded":"2024-07-26T06:14:31.919250+0000","last_fullsized":"2024-07-26T06:14:31.919250+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:19.703607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255025+0000","last_change":"2024-07-26T06:14:29.902772+0000","last_active":"2024-07-26T06:14:32.255025+0000","last_peered":"2024-07-26T06:14:32.255025+0000","last_clean":"2024-07-26T06:14:32.255025+0000","last_became_active":"2024-07-26T06:14:29.902290+0000","last_became_peered":"2024-07-26T06:14:29.902290+0000","last_unstale":"2024-07-26T06:14:32.255025+0000","last_undegraded":"2024-07-26T06:14:32.255025+0000","last_fullsized":"2024-07-26T06:14:32.255025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:05.770984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919382+0000","last_change":"2024-07-26T06:14:30.910393+0000","last_active":"2024-07-26T06:14:31.919382+0000","last_peered":"2024-07-26T06:14:31.919382+0000","last_clean":"2024-07-26T06:14:31.919382+0000","last_became_active":"2024-07-26T06:14:30.910073+0000","last_became_peered":"2024-07-26T06:14:30.910073+0000","last_unstale":"2024-07-26T06:14:31.919382+0000","last_undegraded":"2024-07-26T06:14:31.919382+0000","last_fullsized":"2024-07-26T06:14:31.919382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:34:06.163321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.901630+0000","last_change":"2024-07-26T06:14:29.887700+0000","last_active":"2024-07-26T06:14:30.901630+0000","last_peered":"2024-07-26T06:14:30.901630+0000","last_clean":"2024-07-26T06:14:30.901630+0000","last_became_active":"2024-07-26T06:14:29.886992+0000","last_became_peered":"2024-07-26T06:14:29.886992+0000","last_unstale":"2024-07-26T06:14:30.901630+0000","last_undegraded":"2024-07-26T06:14:30.901630+0000","last_fullsized":"2024-07-26T06:14:30.901630+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:24.378070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.945051+0000","last_change":"2024-07-26T06:14:30.938494+0000","last_active":"2024-07-26T06:14:30.945051+0000","last_peered":"2024-07-26T06:14:30.945051+0000","last_clean":"2024-07-26T06:14:30.945051+0000","last_became_active":"2024-07-26T06:14:30.938181+0000","last_became_peered":"2024-07-26T06:14:30.938181+0000","last_unstale":"2024-07-26T06:14:30.945051+0000","last_undegraded":"2024-07-26T06:14:30.945051+0000","last_fullsized":"2024-07-26T06:14:30.945051+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:28:27.267772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.927128+0000","last_change":"2024-07-26T06:14:29.902112+0000","last_active":"2024-07-26T06:14:30.927128+0000","last_peered":"2024-07-26T06:14:30.927128+0000","last_clean":"2024-07-26T06:14:30.927128+0000","last_became_active":"2024-07-26T06:14:29.901767+0000","last_became_peered":"2024-07-26T06:14:29.901767+0000","last_unstale":"2024-07-26T06:14:30.927128+0000","last_undegraded":"2024-07-26T06:14:30.927128+0000","last_fullsized":"2024-07-26T06:14:30.927128+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:15:44.754272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:31.919404+0000","last_change":"2024-07-26T06:14:30.937605+0000","last_active":"2024-07-26T06:14:31.919404+0000","last_peered":"2024-07-26T06:14:31.919404+0000","last_clean":"2024-07-26T06:14:31.919404+0000","last_became_active":"2024-07-26T06:14:30.935167+0000","last_became_peered":"2024-07-26T06:14:30.935167+0000","last_unstale":"2024-07-26T06:14:31.919404+0000","last_undegraded":"2024-07-26T06:14:31.919404+0000","last_fullsized":"2024-07-26T06:14:31.919404+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:39:32.175666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.943376+0000","last_change":"2024-07-26T06:14:29.902564+0000","last_active":"2024-07-26T06:14:30.943376+0000","last_peered":"2024-07-26T06:14:30.943376+0000","last_clean":"2024-07-26T06:14:30.943376+0000","last_became_active":"2024-07-26T06:14:29.902265+0000","last_became_peered":"2024-07-26T06:14:29.902265+0000","last_unstale":"2024-07-26T06:14:30.943376+0000","last_undegraded":"2024-07-26T06:14:30.943376+0000","last_fullsized":"2024-07-26T06:14:30.943376+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:06:11.025730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.942785+0000","last_change":"2024-07-26T06:14:30.936239+0000","last_active":"2024-07-26T06:14:30.942785+0000","last_peered":"2024-07-26T06:14:30.942785+0000","last_clean":"2024-07-26T06:14:30.942785+0000","last_became_active":"2024-07-26T06:14:30.935650+0000","last_became_peered":"2024-07-26T06:14:30.935650+0000","last_unstale":"2024-07-26T06:14:30.942785+0000","last_undegraded":"2024-07-26T06:14:30.942785+0000","last_fullsized":"2024-07-26T06:14:30.942785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:03.697326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.902468+0000","last_change":"2024-07-26T06:14:29.895723+0000","last_active":"2024-07-26T06:14:30.902468+0000","last_peered":"2024-07-26T06:14:30.902468+0000","last_clean":"2024-07-26T06:14:30.902468+0000","last_became_active":"2024-07-26T06:14:29.895403+0000","last_became_peered":"2024-07-26T06:14:29.895403+0000","last_unstale":"2024-07-26T06:14:30.902468+0000","last_undegraded":"2024-07-26T06:14:30.902468+0000","last_fullsized":"2024-07-26T06:14:30.902468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:18.381616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.931003+0000","last_change":"2024-07-26T06:14:30.923173+0000","last_active":"2024-07-26T06:14:30.931003+0000","last_peered":"2024-07-26T06:14:30.931003+0000","last_clean":"2024-07-26T06:14:30.931003+0000","last_became_active":"2024-07-26T06:14:30.922677+0000","last_became_peered":"2024-07-26T06:14:30.922677+0000","last_unstale":"2024-07-26T06:14:30.931003+0000","last_undegraded":"2024-07-26T06:14:30.931003+0000","last_fullsized":"2024-07-26T06:14:30.931003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:39:32.175666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.917559+0000","last_change":"2024-07-26T06:14:29.890668+0000","last_active":"2024-07-26T06:14:30.917559+0000","last_peered":"2024-07-26T06:14:30.917559+0000","last_clean":"2024-07-26T06:14:30.917559+0000","last_became_active":"2024-07-26T06:14:29.890373+0000","last_became_peered":"2024-07-26T06:14:29.890373+0000","last_unstale":"2024-07-26T06:14:30.917559+0000","last_undegraded":"2024-07-26T06:14:30.917559+0000","last_fullsized":"2024-07-26T06:14:30.917559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:23.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255055+0000","last_change":"2024-07-26T06:14:30.926339+0000","last_active":"2024-07-26T06:14:32.255055+0000","last_peered":"2024-07-26T06:14:32.255055+0000","last_clean":"2024-07-26T06:14:32.255055+0000","last_became_active":"2024-07-26T06:14:30.926006+0000","last_became_peered":"2024-07-26T06:14:30.926006+0000","last_unstale":"2024-07-26T06:14:32.255055+0000","last_undegraded":"2024-07-26T06:14:32.255055+0000","last_fullsized":"2024-07-26T06:14:32.255055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:10:19.889725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.907877+0000","last_change":"2024-07-26T06:14:29.904504+0000","last_active":"2024-07-26T06:14:30.907877+0000","last_peered":"2024-07-26T06:14:30.907877+0000","last_clean":"2024-07-26T06:14:30.907877+0000","last_became_active":"2024-07-26T06:14:29.904005+0000","last_became_peered":"2024-07-26T06:14:29.904005+0000","last_unstale":"2024-07-26T06:14:30.907877+0000","last_undegraded":"2024-07-26T06:14:30.907877+0000","last_fullsized":"2024-07-26T06:14:30.907877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:34:05.152435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.940920+0000","last_change":"2024-07-26T06:14:30.931950+0000","last_active":"2024-07-26T06:14:30.940920+0000","last_peered":"2024-07-26T06:14:30.940920+0000","last_clean":"2024-07-26T06:14:30.940920+0000","last_became_active":"2024-07-26T06:14:30.931673+0000","last_became_peered":"2024-07-26T06:14:30.931673+0000","last_unstale":"2024-07-26T06:14:30.940920+0000","last_undegraded":"2024-07-26T06:14:30.940920+0000","last_fullsized":"2024-07-26T06:14:30.940920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:34:47.647661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.913216+0000","last_change":"2024-07-26T06:14:29.901011+0000","last_active":"2024-07-26T06:14:30.913216+0000","last_peered":"2024-07-26T06:14:30.913216+0000","last_clean":"2024-07-26T06:14:30.913216+0000","last_became_active":"2024-07-26T06:14:29.900703+0000","last_became_peered":"2024-07-26T06:14:29.900703+0000","last_unstale":"2024-07-26T06:14:30.913216+0000","last_undegraded":"2024-07-26T06:14:30.913216+0000","last_fullsized":"2024-07-26T06:14:30.913216+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:18.082903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.935069+0000","last_change":"2024-07-26T06:14:30.918227+0000","last_active":"2024-07-26T06:14:30.935069+0000","last_peered":"2024-07-26T06:14:30.935069+0000","last_clean":"2024-07-26T06:14:30.935069+0000","last_became_active":"2024-07-26T06:14:30.917780+0000","last_became_peered":"2024-07-26T06:14:30.917780+0000","last_unstale":"2024-07-26T06:14:30.935069+0000","last_undegraded":"2024-07-26T06:14:30.935069+0000","last_fullsized":"2024-07-26T06:14:30.935069+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:19:59.788774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.902071+0000","last_change":"2024-07-26T06:14:29.904959+0000","last_active":"2024-07-26T06:14:30.902071+0000","last_peered":"2024-07-26T06:14:30.902071+0000","last_clean":"2024-07-26T06:14:30.902071+0000","last_became_active":"2024-07-26T06:14:29.903801+0000","last_became_peered":"2024-07-26T06:14:29.903801+0000","last_unstale":"2024-07-26T06:14:30.902071+0000","last_undegraded":"2024-07-26T06:14:30.902071+0000","last_fullsized":"2024-07-26T06:14:30.902071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:57:17.278062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.929739+0000","last_change":"2024-07-26T06:14:30.917118+0000","last_active":"2024-07-26T06:14:30.929739+0000","last_peered":"2024-07-26T06:14:30.929739+0000","last_clean":"2024-07-26T06:14:30.929739+0000","last_became_active":"2024-07-26T06:14:30.916644+0000","last_became_peered":"2024-07-26T06:14:30.916644+0000","last_unstale":"2024-07-26T06:14:30.929739+0000","last_undegraded":"2024-07-26T06:14:30.929739+0000","last_fullsized":"2024-07-26T06:14:30.929739+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:20:11.701711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.907132+0000","last_change":"2024-07-26T06:14:29.902632+0000","last_active":"2024-07-26T06:14:30.907132+0000","last_peered":"2024-07-26T06:14:30.907132+0000","last_clean":"2024-07-26T06:14:30.907132+0000","last_became_active":"2024-07-26T06:14:29.902365+0000","last_became_peered":"2024-07-26T06:14:29.902365+0000","last_unstale":"2024-07-26T06:14:30.907132+0000","last_undegraded":"2024-07-26T06:14:30.907132+0000","last_fullsized":"2024-07-26T06:14:30.907132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:37:37.293521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.939193+0000","last_change":"2024-07-26T06:14:30.931640+0000","last_active":"2024-07-26T06:14:30.939193+0000","last_peered":"2024-07-26T06:14:30.939193+0000","last_clean":"2024-07-26T06:14:30.939193+0000","last_became_active":"2024-07-26T06:14:30.931342+0000","last_became_peered":"2024-07-26T06:14:30.931342+0000","last_unstale":"2024-07-26T06:14:30.939193+0000","last_undegraded":"2024-07-26T06:14:30.939193+0000","last_fullsized":"2024-07-26T06:14:30.939193+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:29.855329+0000","last_clean_scrub_stamp":"2024-07-26T06:14:29.855329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:27:38.310414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-07-26T06:14:32.255026+0000","last_change":"2024-07-26T06:14:29.909564+0000","last_active":"2024-07-26T06:14:32.255026+0000","last_peered":"2024-07-26T06:14:32.255026+0000","last_clean":"2024-07-26T06:14:32.255026+0000","last_became_active":"2024-07-26T06:14:29.909199+0000","last_became_peered":"2024-07-26T06:14:29.909199+0000","last_unstale":"2024-07-26T06:14:32.255026+0000","last_undegraded":"2024-07-26T06:14:32.255026+0000","last_fullsized":"2024-07-26T06:14:32.255026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:14:28.844443+0000","last_clean_scrub_stamp":"2024-07-26T06:14:28.844443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:39.524552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":131,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-07-26T06:14:30.921971+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:14:30.921971+0000","last_peered":"2024-07-26T06:14:30.921971+0000","last_clean":"2024-07-26T06:14:30.921971+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:14:30.921971+0000","last_undegraded":"2024-07-26T06:14:30.921971+0000","last_fullsized":"2024-07-26T06:14:30.921971+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509457,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":731671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509458,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":141829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":141885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509459,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509458,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":958464,"data_stored":731581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542161,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":141317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":142847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":352256,"data_stored":134438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:14:35.840 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:14:35.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-07-26T06:14:36.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 -- 172.21.15.163:0/494399371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff588071a70 msgr2=0x7ff5880b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 --2- 172.21.15.163:0/494399371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff588071a70 0x7ff5880b3710 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff580005fd0 tx=0x7ff58002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 -- 172.21.15.163:0/494399371 shutdown_connections 2024-07-26T06:14:36.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 --2- 172.21.15.163:0/494399371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff588071a70 0x7ff5880b3710 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 --2- 172.21.15.163:0/494399371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5880ce4a0 0x7ff588071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 -- 172.21.15.163:0/494399371 >> 172.21.15.163:0/494399371 conn(0x7ff5880b0e30 msgr2=0x7ff5880b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:36.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 -- 172.21.15.163:0/494399371 shutdown_connections 2024-07-26T06:14:36.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.091+0000 7ff58e200640 1 -- 172.21.15.163:0/494399371 wait complete. 2024-07-26T06:14:36.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.093+0000 7ff58e200640 1 Processor -- start 2024-07-26T06:14:36.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.093+0000 7ff58e200640 1 -- start start 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff588071a70 0x7ff5881f1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 0x7ff5881410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5881556f0 0x7ff5881e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5880b4020 con 0x7ff5880ce4a0 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff5880b3ea0 con 0x7ff5881556f0 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff5880b41a0 con 0x7ff588071a70 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff57fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 0x7ff5881410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff57fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 0x7ff5881410a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40508/0 (socket says 172.21.15.163:40508) 2024-07-26T06:14:36.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff57fe00640 1 -- 172.21.15.163:0/3138383167 learned_addr learned my addr 172.21.15.163:0/3138383167 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff57fe00640 1 -- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff588071a70 msgr2=0x7ff5881f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58ce00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff588071a70 0x7ff5881f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.094+0000 7ff58d800640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5881556f0 0x7ff5881e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff57fe00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff588071a70 0x7ff5881f1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff57fe00640 1 -- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5881556f0 msgr2=0x7ff5881e38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff57fe00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5881556f0 0x7ff5881e38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff57fe00640 1 -- 172.21.15.163:0/3138383167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff588155e90 con 0x7ff5880ce4a0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff58ce00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff588071a70 0x7ff5881f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff57fe00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 0x7ff5881410a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff57000c970 tx=0x7ff57000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570014070 con 0x7ff5880ce4a0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff570007860 con 0x7ff5880ce4a0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570007cb0 con 0x7ff5880ce4a0 2024-07-26T06:14:36.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff588156170 con 0x7ff5880ce4a0 2024-07-26T06:14:36.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.095+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff588156730 con 0x7ff5880ce4a0 2024-07-26T06:14:36.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.096+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff588156930 con 0x7ff5880ce4a0 2024-07-26T06:14:36.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.098+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff570017070 con 0x7ff5880ce4a0 2024-07-26T06:14:36.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.099+0000 7ff585600640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff5640762d0 0x7ff564078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.099+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7ff570097a60 con 0x7ff5880ce4a0 2024-07-26T06:14:36.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.099+0000 7ff58ce00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff5640762d0 0x7ff564078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.100+0000 7ff58ce00640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff5640762d0 0x7ff564078790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff5880af7a0 tx=0x7ff578009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.103+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff588156930 con 0x7ff5880ce4a0 2024-07-26T06:14:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.369+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff5880b4730 con 0x7ff5880ce4a0 2024-07-26T06:14:36.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.373+0000 7ff585600640 1 -- 172.21.15.163:0/3138383167 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) ==== 190+0+147 (secure 0 0 0) 0x7ff5700613f0 con 0x7ff5880ce4a0 2024-07-26T06:14:36.374 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:14:36.374 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:14:36.374 INFO:teuthology.orchestra.run.smithi163.stdout: caps mds = "allow" 2024-07-26T06:14:36.374 INFO:teuthology.orchestra.run.smithi163.stdout: caps mon = "allow r" 2024-07-26T06:14:36.374 INFO:teuthology.orchestra.run.smithi163.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-07-26T06:14:36.374 INFO:teuthology.orchestra.run.smithi163.stderr:updated caps for client.0 2024-07-26T06:14:36.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.375+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff5640762d0 msgr2=0x7ff564078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.375+0000 7ff58e200640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff5640762d0 0x7ff564078790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff5880af7a0 tx=0x7ff578009040 comp rx=0 tx=0).stop 2024-07-26T06:14:36.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.375+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 msgr2=0x7ff5881410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.375+0000 7ff58e200640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 0x7ff5881410a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff57000c970 tx=0x7ff57000ce30 comp rx=0 tx=0).stop 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 shutdown_connections 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff5640762d0 0x7ff564078790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5881556f0 0x7ff5881e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5880ce4a0 0x7ff5881410a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 --2- 172.21.15.163:0/3138383167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff588071a70 0x7ff5881f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 >> 172.21.15.163:0/3138383167 conn(0x7ff5880b0e30 msgr2=0x7ff58815b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 shutdown_connections 2024-07-26T06:14:36.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.376+0000 7ff58e200640 1 -- 172.21.15.163:0/3138383167 wait complete. 2024-07-26T06:14:36.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 -- 172.21.15.163:0/3651756800 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 msgr2=0x7f47a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 --2- 172.21.15.163:0/3651756800 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47a0003e20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00c7a0 tx=0x7f47a0005830 comp rx=0 tx=0).stop 2024-07-26T06:14:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 -- 172.21.15.163:0/3651756800 shutdown_connections 2024-07-26T06:14:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 --2- 172.21.15.163:0/3651756800 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47a0003e20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 -- 172.21.15.163:0/3651756800 >> 172.21.15.163:0/3651756800 conn(0x7f47bc0b1ee0 msgr2=0x7f47bc0b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 -- 172.21.15.163:0/3651756800 shutdown_connections 2024-07-26T06:14:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.597+0000 7f47c0e00640 1 -- 172.21.15.163:0/3651756800 wait complete. 2024-07-26T06:14:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.599+0000 7f47c0e00640 1 Processor -- start 2024-07-26T06:14:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.599+0000 7f47c0e00640 1 -- start start 2024-07-26T06:14:36.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47c0e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47bc147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47c0e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47bc14f2b0 0x7f47bc1509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47c0e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47bc150f20 0x7f47bc0b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47c0e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc0cf6d0 con 0x7f47a0003a00 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47c0e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc06c5b0 con 0x7f47bc14f2b0 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47c0e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f47bc0cf850 con 0x7f47bc150f20 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47bb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47bc147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47bb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47bc147a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40520/0 (socket says 172.21.15.163:40520) 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.600+0000 7f47bb400640 1 -- 172.21.15.163:0/2475331951 learned_addr learned my addr 172.21.15.163:0/2475331951 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47bb400640 1 -- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47bc150f20 msgr2=0x7f47bc0b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47bb400640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47bc150f20 0x7f47bc0b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47bb400640 1 -- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47bc14f2b0 msgr2=0x7f47bc1509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47bb400640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47bc14f2b0 0x7f47bc1509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47bb400640 1 -- 172.21.15.163:0/2475331951 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47bc1514a0 con 0x7f47a0003a00 2024-07-26T06:14:36.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47bb400640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47bc147a20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f47ac000f80 tx=0x7f47ac00c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:36.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac003b70 con 0x7f47a0003a00 2024-07-26T06:14:36.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47ac03a070 con 0x7f47a0003a00 2024-07-26T06:14:36.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac034470 con 0x7f47a0003a00 2024-07-26T06:14:36.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47bc151780 con 0x7f47a0003a00 2024-07-26T06:14:36.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.601+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47bc152020 con 0x7f47a0003a00 2024-07-26T06:14:36.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.603+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47bc15b700 con 0x7f47a0003a00 2024-07-26T06:14:36.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.604+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f47ac03c070 con 0x7f47a0003a00 2024-07-26T06:14:36.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.605+0000 7f47b3e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f47980762d0 0x7f4798078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:36.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.605+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f47ac0b77b0 con 0x7f47a0003a00 2024-07-26T06:14:36.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.605+0000 7f47b2a00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f47980762d0 0x7f4798078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:36.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.607+0000 7f47b2a00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f47980762d0 0x7f4798078790 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f47a4006fd0 tx=0x7f47a4006910 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.609+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47bc06c890 con 0x7f47a0003a00 2024-07-26T06:14:36.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.857+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47bc151950 con 0x7f47a0003a00 2024-07-26T06:14:36.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.858+0000 7f47b3e00640 1 -- 172.21.15.163:0/2475331951 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5229 (secure 0 0 0) 0x7f47ac0810f0 con 0x7f47a0003a00 2024-07-26T06:14:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:36.860 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":16,"btime":"2024-07-26T06:14:33:929585+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:14:30.337611+0000","modified":"2024-07-26T06:14:33.428686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.163:6835/941040757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":941040757},{"type":"v1","addr":"172.21.15.163:6835","nonce":941040757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-07-26T06:14:36.860 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 16 2024-07-26T06:14:36.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.861+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f47980762d0 msgr2=0x7f4798078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.861+0000 7f47c0e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f47980762d0 0x7f4798078790 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f47a4006fd0 tx=0x7f47a4006910 comp rx=0 tx=0).stop 2024-07-26T06:14:36.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.861+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 msgr2=0x7f47bc147a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.861+0000 7f47c0e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47bc147a20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f47ac000f80 tx=0x7f47ac00c4a0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 shutdown_connections 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f47980762d0 0x7f4798078790 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47bc150f20 0x7f47bc0b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47bc14f2b0 0x7f47bc1509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 --2- 172.21.15.163:0/2475331951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47a0003a00 0x7f47bc147a20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 >> 172.21.15.163:0/2475331951 conn(0x7f47bc0b1ee0 msgr2=0x7f47bc0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:36.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 shutdown_connections 2024-07-26T06:14:36.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:36.862+0000 7f47c0e00640 1 -- 172.21.15.163:0/2475331951 wait complete. 2024-07-26T06:14:37.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:14:30.337611+0000', 'modified': '2024-07-26T06:14:33.428686+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4845}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4845': {'gid': 4845, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.163:6835/941040757', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6833', 'nonce': 941040757}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 941040757}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4845, 'qdb_cluster': [4845]} 2024-07-26T06:14:37.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:14:37.303 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-07-26T06:14:37.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 -- 172.21.15.163:0/2176170607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cbad0 msgr2=0x7f88bc0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2176170607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cbad0 0x7f88bc0cd530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0b1bc0 tx=0x7f88ac02aa30 comp rx=0 tx=0).stop 2024-07-26T06:14:37.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 -- 172.21.15.163:0/2176170607 shutdown_connections 2024-07-26T06:14:37.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2176170607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88bc0cd000 0x7f88bc15ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2176170607 >> v2:172.21.15.164:3300/0 conn(0x7f88bc0b4420 0x7f88bc0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2176170607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cbad0 0x7f88bc0cd530 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 -- 172.21.15.163:0/2176170607 >> 172.21.15.163:0/2176170607 conn(0x7f88bc0b0e30 msgr2=0x7f88bc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 -- 172.21.15.163:0/2176170607 shutdown_connections 2024-07-26T06:14:37.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.519+0000 7f88c1e00640 1 -- 172.21.15.163:0/2176170607 wait complete. 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.521+0000 7f88c1e00640 1 Processor -- start 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.521+0000 7f88c1e00640 1 -- start start 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88bc0b4420 0x7f88bc14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 0x7f88bc14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88bc143b40 0x7f88bc151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88c1e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc1594f0 con 0x7f88bc0cd000 2024-07-26T06:14:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88c1e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc159370 con 0x7f88bc143b40 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88c1e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88bc159670 con 0x7f88bc0b4420 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88bbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 0x7f88bc14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88bbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 0x7f88bc14cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40536/0 (socket says 172.21.15.163:40536) 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.522+0000 7f88bbe00640 1 -- 172.21.15.163:0/2405104592 learned_addr learned my addr 172.21.15.163:0/2405104592 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88c0a00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88bc0b4420 0x7f88bc14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88c1400640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88bc143b40 0x7f88bc151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88bbe00640 1 -- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88bc0b4420 msgr2=0x7f88bc14a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88bbe00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88bc0b4420 0x7f88bc14a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88bbe00640 1 -- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88bc143b40 msgr2=0x7f88bc151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88bbe00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88bc143b40 0x7f88bc151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88bbe00640 1 -- 172.21.15.163:0/2405104592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc147cd0 con 0x7f88bc0cd000 2024-07-26T06:14:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88bbe00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 0x7f88bc14cee0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f88a400d770 tx=0x7f88a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.523+0000 7f88b8c00640 1 -- 172.21.15.163:0/2405104592 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4017070 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.524+0000 7f88b8c00640 1 -- 172.21.15.163:0/2405104592 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88a4009d20 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.524+0000 7f88b8c00640 1 -- 172.21.15.163:0/2405104592 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a40106c0 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.524+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88bc147f70 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.524+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88bc14ac90 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.526+0000 7f88b8c00640 1 -- 172.21.15.163:0/2405104592 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f88a401a070 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.528+0000 7f88b8c00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88980762d0 0x7f8898078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.528+0000 7f88b8c00640 1 -- 172.21.15.163:0/2405104592 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f88a4097940 con 0x7f88bc0cd000 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.528+0000 7f88c6220740 1 Processor -- start 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.528+0000 7f88c0a00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88980762d0 0x7f8898078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.528+0000 7f88c6220740 1 -- start start 2024-07-26T06:14:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.529+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.529+0000 7f88c6220740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 0x5624de9aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.529+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de96be50 0x5624de964fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.529+0000 7f88c6220740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5624de8813f0 con 0x5624de9b08d0 2024-07-26T06:14:37.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.529+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5624de8dc5c0 con 0x5624de96be50 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.529+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5624de8cbb60 con 0x5624de9b1560 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 0x5624de9aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 0x5624de9aeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40542/0 (socket says 172.21.15.163:40542) 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 -- 172.21.15.163:0/2745145348 learned_addr learned my addr 172.21.15.163:0/2745145348 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88c0a00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88980762d0 0x7f8898078790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88ac035000 tx=0x7f88ac004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88c0a00640 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88c1400640 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de96be50 0x5624de964fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 -- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 msgr2=0x5624de9b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 -- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de96be50 msgr2=0x5624de964fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de96be50 0x5624de964fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 -- 172.21.15.163:0/2745145348 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624de630850 con 0x5624de9b08d0 2024-07-26T06:14:37.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88c1400640 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de96be50 0x5624de964fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.530+0000 7f88bbe00640 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 0x5624de9aeea0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88a405d0f0 tx=0x7f88a4069210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4f1e04b53d053a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88b1200640 1 -- 172.21.15.163:0/2745145348 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4021040 con 0x5624de9b08d0 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88b1200640 1 -- 172.21.15.163:0/2745145348 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88a4060b00 con 0x5624de9b08d0 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88b1200640 1 -- 172.21.15.163:0/2745145348 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a405ffc0 con 0x5624de9b08d0 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 -- 172.21.15.163:0/2745145348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 msgr2=0x5624de9aeea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 0x5624de9aeea0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88a405d0f0 tx=0x7f88a4069210 comp rx=0 tx=0).stop 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 -- 172.21.15.163:0/2745145348 shutdown_connections 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de96be50 0x5624de964fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b08d0 0x5624de9aeea0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 --2- 172.21.15.163:0/2745145348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.531+0000 7f88c6220740 1 -- 172.21.15.163:0/2745145348 >> 172.21.15.163:0/2745145348 conn(0x5624de9671a0 msgr2=0x5624de9af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.532+0000 7f88c6220740 1 -- 172.21.15.163:0/2745145348 shutdown_connections 2024-07-26T06:14:37.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.532+0000 7f88c6220740 1 -- 172.21.15.163:0/2745145348 wait complete. 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.532+0000 7f88c6220740 1 -- start start 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.532+0000 7f88c6220740 1 Processor -- start 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.533+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.533+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de993960 0x5624de993d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.533+0000 7f88c6220740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 0x5624de9aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c6220740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5624de874210 con 0x5624de9a9c60 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5624de8cfba0 con 0x5624de993960 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5624de9afce0 con 0x5624de9b1560 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c6220740 10 client.0 _ll_register_callbacks cb 0x5624de8748a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 0x5624de9aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 0x5624de9aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40552/0 (socket says 172.21.15.163:40552) 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 -- 172.21.15.163:0/1843613337 learned_addr learned my addr 172.21.15.163:0/1843613337 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c0a00640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88bbe00640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de993960 0x5624de993d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 -- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 msgr2=0x5624de9a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9a8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 -- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de993960 msgr2=0x5624de993d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de993960 0x5624de993d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88c1400640 1 -- 172.21.15.163:0/1843613337 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624de6a76b0 con 0x5624de9a9c60 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.534+0000 7f88bbe00640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de993960 0x5624de993d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88c1400640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 0x5624de9aa0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f88b400dd50 tx=0x7f88b400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3423687c30b4a245 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88a9600640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88a9600640 1 -- 172.21.15.163:0/1843613337 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4004020 con 0x5624de9a9c60 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88a9600640 1 -- 172.21.15.163:0/1843613337 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f88b40041c0 con 0x5624de9a9c60 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5624de8de510 con 0x5624de9a9c60 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88a9600640 1 -- 172.21.15.163:0/1843613337 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f88b40043c0 con 0x5624de9a9c60 2024-07-26T06:14:37.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88c6220740 10 client.4995 fetch_fsmap learned FSMap version 16 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x5624de9a9300 con 0x5624de9a9c60 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.535+0000 7f88a9600640 1 -- 172.21.15.163:0/1843613337 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x5624de8de510 con 0x5624de9a9c60 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88a9600640 1 -- 172.21.15.163:0/1843613337 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 16) ==== 3502+0+0 (secure 0 0 0) 0x7f88b4021050 con 0x5624de9a9c60 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c6220740 10 client.4995 fetch_fsmap finished waiting for FSMap version 16 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c6220740 10 client.4995 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c6220740 20 client.4995 populate_metadata read hostname 'smithi163' 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c6220740 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de9a7e80 0x5624de9cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c6220740 4 client.4995 mds_command: new command op to 4845 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 --> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624de8d9fb0 con 0x5624de9a7e80 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.536+0000 7f88c0a00640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de9a7e80 0x5624de9cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.537+0000 7f88c0a00640 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de9a7e80 0x5624de9cdd90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee67a1c671c11d8f server_cookie=e207badd4aded4de in_seq=0 out_seq=0 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.537+0000 7f88a9600640 10 client.4995 ms_handle_connect on v2:172.21.15.163:6833/941040757 2024-07-26T06:14:37.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.537+0000 7f88a9600640 1 -- 172.21.15.163:0/1843613337 <== mds.0 v2:172.21.15.163:6833/941040757 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5624de8d9fb0 con 0x5624de9a7e80 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.537+0000 7f88a9600640 10 client.4995 handle_command_reply: tid=0 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 client.4995 shutdown 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 msgr2=0x5624de9aa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 0x5624de9aa0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f88b400dd50 tx=0x7f88b400f730 comp rx=0 tx=0).stop 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 shutdown_connections 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de9a7e80 0x5624de9cdd90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9a9c60 0x5624de9aa0e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de993960 0x5624de993d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 --2- 172.21.15.163:0/1843613337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88a9600640 0 client.4995 ms_handle_reset on v2:172.21.15.163:6833/941040757 2024-07-26T06:14:37.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 >> 172.21.15.163:0/1843613337 conn(0x5624de96be50 msgr2=0x5624de9b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 shutdown_connections 2024-07-26T06:14:37.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 1 -- 172.21.15.163:0/1843613337 wait complete. 2024-07-26T06:14:37.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.538+0000 7f88c6220740 20 client.4995 trim_cache size 0 max 16384 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.559+0000 7f88c6220740 1 Processor -- start 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.559+0000 7f88c6220740 1 -- start start 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c6220740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 0x5624de9bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c6220740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc200500 con 0x5624de993960 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc1ff580 con 0x5624de9a9c60 2024-07-26T06:14:37.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88bc1549a0 con 0x5624de9b1560 2024-07-26T06:14:37.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.560+0000 7f88c0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 0x5624de9bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 0x5624de9bf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40558/0 (socket says 172.21.15.163:40558) 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 -- 172.21.15.163:0/3608523894 learned_addr learned my addr 172.21.15.163:0/3608523894 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88bbe00640 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c1400640 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 -- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 msgr2=0x5624de9b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 -- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 msgr2=0x5624de9bff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 -- 172.21.15.163:0/3608523894 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624de96bb80 con 0x5624de993960 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88bbe00640 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.561+0000 7f88c0a00640 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 0x5624de9bf9d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02b020 tx=0x7f88ac03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb98f1cffdf872a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.562+0000 7f88abe00640 1 -- 172.21.15.163:0/3608523894 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ac056020 con 0x5624de993960 2024-07-26T06:14:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.562+0000 7f88abe00640 1 -- 172.21.15.163:0/3608523894 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88ac0458f0 con 0x5624de993960 2024-07-26T06:14:37.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.562+0000 7f88abe00640 1 -- 172.21.15.163:0/3608523894 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ac033050 con 0x5624de993960 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.562+0000 7f88c6220740 1 -- 172.21.15.163:0/3608523894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 msgr2=0x5624de9bf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.562+0000 7f88c6220740 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 0x5624de9bf9d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02b020 tx=0x7f88ac03d000 comp rx=0 tx=0).stop 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 -- 172.21.15.163:0/3608523894 shutdown_connections 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de9b1560 0x5624de9b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 --2- 172.21.15.163:0/3608523894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de993960 0x5624de9bf9d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 -- 172.21.15.163:0/3608523894 >> 172.21.15.163:0/3608523894 conn(0x5624de96be50 msgr2=0x5624de960870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 -- 172.21.15.163:0/3608523894 shutdown_connections 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 -- 172.21.15.163:0/3608523894 wait complete. 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.563+0000 7f88c6220740 1 -- start start 2024-07-26T06:14:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.564+0000 7f88c6220740 1 Processor -- start 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.564+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de993960 0x5624dea2ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c6220740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c6220740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 0x5624de9bff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c6220740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5624de994ca0 con 0x5624de9b1560 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc1fef30 con 0x5624de9a9c60 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c6220740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5624de9a7880 con 0x5624de993960 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c6220740 10 client.0 _ll_register_callbacks cb 0x7f88b400d880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 0x5624de9bff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 0x5624de9bff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40566/0 (socket says 172.21.15.163:40566) 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 -- 172.21.15.163:0/3033992177 learned_addr learned my addr 172.21.15.163:0/3033992177 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 -- 172.21.15.163:0/3033992177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de993960 msgr2=0x5624dea2ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de993960 0x5624dea2ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 -- 172.21.15.163:0/3033992177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 msgr2=0x5624de9bf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.565+0000 7f88c1400640 1 -- 172.21.15.163:0/3033992177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624dea2e460 con 0x5624de9b1560 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c1400640 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 0x5624de9bff10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x5624de96bd50 tx=0x7f88b4011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be76ab9ec822c985 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88a9600640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88a9600640 1 -- 172.21.15.163:0/3033992177 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b400fc30 con 0x5624de9b1560 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88a9600640 1 -- 172.21.15.163:0/3033992177 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f88b4023680 con 0x5624de9b1560 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88ac0458f0 con 0x5624de9b1560 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88a9600640 1 -- 172.21.15.163:0/3033992177 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f88b4033050 con 0x5624de9b1560 2024-07-26T06:14:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 10 client.5001 fetch_fsmap learned FSMap version 16 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x5624dea2e1a0 con 0x5624de9b1560 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88a9600640 1 -- 172.21.15.163:0/3033992177 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88ac0458f0 con 0x5624de9b1560 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88a9600640 1 -- 172.21.15.163:0/3033992177 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 16) ==== 3502+0+0 (secure 0 0 0) 0x7f88b4036020 con 0x5624de9b1560 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 10 client.5001 fetch_fsmap finished waiting for FSMap version 16 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 10 client.5001 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 20 client.5001 populate_metadata read hostname 'smithi163' 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de990920 0x5624de990d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.566+0000 7f88c6220740 4 client.5001 mds_command: new command op to 4845 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.567+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 --> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] -- command(tid 0: {"prefix": "status"}) -- 0x5624de9ae860 con 0x5624de990920 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.567+0000 7f88c0a00640 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de990920 0x5624de990d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.567+0000 7f88c0a00640 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de990920 0x5624de990d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a6a5ddb30c7e7b4 server_cookie=3cd13c6ea66906c5 in_seq=0 out_seq=0 2024-07-26T06:14:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.567+0000 7f88a9600640 10 client.5001 ms_handle_connect on v2:172.21.15.163:6833/941040757 2024-07-26T06:14:37.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88a9600640 1 -- 172.21.15.163:0/3033992177 <== mds.0 v2:172.21.15.163:6833/941040757 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5624de9ae860 con 0x5624de990920 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88a9600640 10 client.5001 handle_command_reply: tid=0 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 client.5001 shutdown 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 msgr2=0x5624de9bff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 0x5624de9bff10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x5624de96bd50 tx=0x7f88b4011eb0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 shutdown_connections 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:6833/941040757,v1:172.21.15.163:6835/941040757] conn(0x5624de990920 0x5624de990d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624de9b1560 0x5624de9bff10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624de9a9c60 0x5624de9bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 --2- 172.21.15.163:0/3033992177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5624de993960 0x5624dea2ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 >> 172.21.15.163:0/3033992177 conn(0x5624de96be50 msgr2=0x5624de96ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88a9600640 0 client.5001 ms_handle_reset on v2:172.21.15.163:6833/941040757 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 shutdown_connections 2024-07-26T06:14:37.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 1 -- 172.21.15.163:0/3033992177 wait complete. 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.568+0000 7f88c6220740 20 client.5001 trim_cache size 0 max 16384 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 4845, 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 6.6759578360000003, 2024-07-26T06:14:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 16, 2024-07-26T06:14:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 34, 2024-07-26T06:14:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 34, 2024-07-26T06:14:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 19.513871567999999 2024-07-26T06:14:37.574 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:14:37.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88980762d0 msgr2=0x7f8898078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88980762d0 0x7f8898078790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88ac035000 tx=0x7f88ac004ab0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 msgr2=0x7f88bc14cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 0x7f88bc14cee0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f88a400d770 tx=0x7f88a400dc30 comp rx=0 tx=0).stop 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 shutdown_connections 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88980762d0 0x7f8898078790 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88bc143b40 0x7f88bc151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88bc0cd000 0x7f88bc14cee0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 --2- 172.21.15.163:0/2405104592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88bc0b4420 0x7f88bc14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 >> 172.21.15.163:0/2405104592 conn(0x7f88bc0b0e30 msgr2=0x7f88bc0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 shutdown_connections 2024-07-26T06:14:37.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.569+0000 7f88c1e00640 1 -- 172.21.15.163:0/2405104592 wait complete. 2024-07-26T06:14:37.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:14:37.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 -- 172.21.15.163:0/1656632040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0440052e0 msgr2=0x7ff04400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 --2- 172.21.15.163:0/1656632040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0440052e0 0x7ff04400b9c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff05800c590 tx=0x7ff044005830 comp rx=0 tx=0).stop 2024-07-26T06:14:37.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 -- 172.21.15.163:0/1656632040 shutdown_connections 2024-07-26T06:14:37.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 --2- 172.21.15.163:0/1656632040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0440052e0 0x7ff04400b9c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 -- 172.21.15.163:0/1656632040 >> 172.21.15.163:0/1656632040 conn(0x7ff0600cfda0 msgr2=0x7ff0600d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:37.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 -- 172.21.15.163:0/1656632040 shutdown_connections 2024-07-26T06:14:37.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.780+0000 7ff05f200640 1 -- 172.21.15.163:0/1656632040 wait complete. 2024-07-26T06:14:37.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.782+0000 7ff05f200640 1 Processor -- start 2024-07-26T06:14:37.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.782+0000 7ff05f200640 1 -- start start 2024-07-26T06:14:37.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0440052e0 0x7ff060154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0601583f0 0x7ff0600cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 0x7ff060155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff0600ce470 con 0x7ff0601553d0 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0600ce2f0 con 0x7ff0601583f0 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff0600ce5f0 con 0x7ff0440052e0 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 0x7ff060155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 0x7ff060155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40596/0 (socket says 172.21.15.163:40596) 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.783+0000 7ff05e800640 1 -- 172.21.15.163:0/115057704 learned_addr learned my addr 172.21.15.163:0/115057704 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05de00640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0440052e0 0x7ff060154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff057e00640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0601583f0 0x7ff0600cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05e800640 1 -- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0440052e0 msgr2=0x7ff060154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05e800640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0440052e0 0x7ff060154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05e800640 1 -- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0601583f0 msgr2=0x7ff0600cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:37.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05e800640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0601583f0 0x7ff0600cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:37.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05e800640 1 -- 172.21.15.163:0/115057704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0600cb520 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05e800640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 0x7ff060155850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff05802b430 tx=0x7ff058003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058039070 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff05802bca0 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.784+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0600cb800 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.785+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058006330 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.785+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0600cbdc0 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.786+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0600b6840 con 0x7ff0601553d0 2024-07-26T06:14:37.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.787+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff05803c070 con 0x7ff0601553d0 2024-07-26T06:14:37.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.788+0000 7ff056000640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff03c0762d0 0x7ff03c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:37.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.789+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7ff0580b71b0 con 0x7ff0601553d0 2024-07-26T06:14:37.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.789+0000 7ff05de00640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff03c0762d0 0x7ff03c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:37.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.790+0000 7ff05de00640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff03c0762d0 0x7ff03c078790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff050009a20 tx=0x7ff050005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:37.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:37.793+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff058080b00 con 0x7ff0601553d0 2024-07-26T06:14:38.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.037+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff0600ceca0 con 0x7ff0601553d0 2024-07-26T06:14:38.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.038+0000 7ff056000640 1 -- 172.21.15.163:0/115057704 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7ff058080400 con 0x7ff0601553d0 2024-07-26T06:14:38.039 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:14:38.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff03c0762d0 msgr2=0x7ff03c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:38.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff03c0762d0 0x7ff03c078790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff050009a20 tx=0x7ff050005f90 comp rx=0 tx=0).stop 2024-07-26T06:14:38.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 msgr2=0x7ff060155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:38.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 0x7ff060155850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff05802b430 tx=0x7ff058003ba0 comp rx=0 tx=0).stop 2024-07-26T06:14:38.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 shutdown_connections 2024-07-26T06:14:38.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff03c0762d0 0x7ff03c078790 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:38.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0601553d0 0x7ff060155850 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:38.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0601583f0 0x7ff0600cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:38.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 --2- 172.21.15.163:0/115057704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0440052e0 0x7ff060154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:38.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 >> 172.21.15.163:0/115057704 conn(0x7ff0600cfda0 msgr2=0x7ff0600d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:38.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 shutdown_connections 2024-07-26T06:14:38.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38.041+0000 7ff05f200640 1 -- 172.21.15.163:0/115057704 wait complete. 2024-07-26T06:14:38.060 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:14:38.060 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:14:38.060 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:14:38.060 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:14:38.060 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:14:38.061 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:14:38.061 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:14:38.061 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:14:38.061 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:14:38.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.061 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:14:38.112 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:14:38.112 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:38.113 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42527sec preferred_lft 42527sec 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:14:38.114 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:14:38.115 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.115 INFO:teuthology.orchestra.run.smithi163.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:14:38.115 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether c2:4f:6d:14:5d:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:14:38.115 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::c04f:6dff:fe14:5d96/64 scope link 2024-07-26T06:14:38.115 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:38.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.116 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:14:38.191 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:14:38.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.195 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:14:38.249 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:14:38.249 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:14:38.249 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:14:38.249 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:14:38.249 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:14:38.250 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:14:38.250 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:14:38.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:14:38.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:14:38.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:14:38.250 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:14:38.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:38.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:38.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.460 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:14:38.507 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:14:38.507 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:14:38.508 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:14:38.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:14:38.589 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:14:38.589 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:14:38.642 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:14:38.643 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:14:38.643 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:14:38.691 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:14:38.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.692 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:14:38.815 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:14:38.815 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:14:38.815 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:14:38.839 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:14:38.839 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:14:38.840 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:14:38.840 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:38.840 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:14:38.867 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:14:38.870 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:14:38.870 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:14:38.870 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:14:38.870 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:14:38.870 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:14:38.875 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:14:38.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:38.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:14:38.957 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:39.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:39.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:39.084 INFO:teuthology.orchestra.run.smithi163.stdout:client4771 2024-07-26T06:14:39.085 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:39.085 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:39.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 -- 172.21.15.163:0/2979883913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff408159380 msgr2=0x7ff4080b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 --2- 172.21.15.163:0/2979883913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff408159380 0x7ff4080b2f90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00a940 tx=0x7ff3fc02b850 comp rx=0 tx=0).stop 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 -- 172.21.15.163:0/2979883913 shutdown_connections 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 --2- 172.21.15.163:0/2979883913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff408159380 0x7ff4080b2f90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 -- 172.21.15.163:0/2979883913 >> 172.21.15.163:0/2979883913 conn(0x7ff40806d420 msgr2=0x7ff40806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 -- 172.21.15.163:0/2979883913 shutdown_connections 2024-07-26T06:14:39.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.365+0000 7ff40f400640 1 -- 172.21.15.163:0/2979883913 wait complete. 2024-07-26T06:14:39.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.367+0000 7ff40f400640 1 Processor -- start 2024-07-26T06:14:39.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.367+0000 7ff40f400640 1 -- start start 2024-07-26T06:14:39.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 0x7ff4081523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:39.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40f400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4080b40a0 0x7ff408151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:39.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40f400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff408159380 0x7ff408153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:39.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40f400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff4080cd170 con 0x7ff40806c650 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40f400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff4080ccff0 con 0x7ff408159380 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40f400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4080cd2f0 con 0x7ff4080b40a0 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 0x7ff4081523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 0x7ff4081523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40616/0 (socket says 172.21.15.163:40616) 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 -- 172.21.15.163:0/1829831151 learned_addr learned my addr 172.21.15.163:0/1829831151 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40d400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff408159380 0x7ff408153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff407e00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4080b40a0 0x7ff408151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 -- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4080b40a0 msgr2=0x7ff408151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4080b40a0 0x7ff408151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 -- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff408159380 msgr2=0x7ff408153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff408159380 0x7ff408153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40ca00640 1 -- 172.21.15.163:0/1829831151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40815bbf0 con 0x7ff40806c650 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.368+0000 7ff40d400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff408159380 0x7ff408153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:39.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.369+0000 7ff40ca00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 0x7ff4081523e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc02be40 tx=0x7ff3fc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:39.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.369+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc041070 con 0x7ff40806c650 2024-07-26T06:14:39.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.369+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3fc005020 con 0x7ff40806c650 2024-07-26T06:14:39.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.369+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc005470 con 0x7ff40806c650 2024-07-26T06:14:39.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.369+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff408152b50 con 0x7ff40806c650 2024-07-26T06:14:39.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.369+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff408153110 con 0x7ff40806c650 2024-07-26T06:14:39.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.370+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4080b2f90 con 0x7ff40806c650 2024-07-26T06:14:39.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.371+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff3fc03b040 con 0x7ff40806c650 2024-07-26T06:14:39.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.373+0000 7ff404c00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff3e8076300 0x7ff3e80787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:39.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.373+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7ff3fc0b83e0 con 0x7ff40806c650 2024-07-26T06:14:39.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.374+0000 7ff407e00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff3e8076300 0x7ff3e80787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:39.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.375+0000 7ff407e00640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff3e8076300 0x7ff3e80787c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4007fd0 tx=0x7ff3f4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:39.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.378+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff3fc081d30 con 0x7ff40806c650 2024-07-26T06:14:39.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.625+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff4080b3870 con 0x7ff40806c650 2024-07-26T06:14:39.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.626+0000 7ff404c00640 1 -- 172.21.15.163:0/1829831151 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7ff3fc081630 con 0x7ff40806c650 2024-07-26T06:14:39.626 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:14:39.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff3e8076300 msgr2=0x7ff3e80787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:39.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff3e8076300 0x7ff3e80787c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4007fd0 tx=0x7ff3f4009040 comp rx=0 tx=0).stop 2024-07-26T06:14:39.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 msgr2=0x7ff4081523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:39.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 0x7ff4081523e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc02be40 tx=0x7ff3fc003c10 comp rx=0 tx=0).stop 2024-07-26T06:14:39.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 shutdown_connections 2024-07-26T06:14:39.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff3e8076300 0x7ff3e80787c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff408159380 0x7ff408153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4080b40a0 0x7ff408151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 --2- 172.21.15.163:0/1829831151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff40806c650 0x7ff4081523e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:39.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 >> 172.21.15.163:0/1829831151 conn(0x7ff40806d420 msgr2=0x7ff4080cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:39.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 shutdown_connections 2024-07-26T06:14:39.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:39.629+0000 7ff40f400640 1 -- 172.21.15.163:0/1829831151 wait complete. 2024-07-26T06:14:39.649 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:14:39.649 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:14:39.649 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:14:39.649 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:14:39.649 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:14:39.649 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:14:39.650 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:14:39.650 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:14:39.650 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:14:39.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:39.650 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:14:39.710 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:14:39.710 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:14:39.710 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:14:39.710 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.710 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42503sec preferred_lft 42503sec 2024-07-26T06:14:39.711 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:14:39.712 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::58b1:99ff:fee0:4877/64 scope link 2024-07-26T06:14:39.713 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:14:39.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:39.714 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:14:39.796 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:14:39.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:39.800 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:14:39.853 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:14:39.854 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:14:39.854 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:14:39.854 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:14:39.854 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:14:39.855 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:14:39.855 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:14:39.855 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:14:39.855 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:14:39.855 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:14:39.855 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:14:39.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:14:40.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:40.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:40.066 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:14:40.119 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:14:40.119 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:14:40.119 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:14:40.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:40.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:14:40.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:14:40.199 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:14:40.252 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:14:40.252 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:14:40.253 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:14:40.302 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:14:40.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:40.303 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:14:40.439 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:14:40.439 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:14:40.439 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:40.463 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:14:40.463 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:14:40.463 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:40.463 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:40.463 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:40.491 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:14:40.491 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:14:40.492 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:14:40.492 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:14:40.492 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:14:40.492 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:14:40.492 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:14:40.492 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:14:40.494 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:14:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:14:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:14:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:14:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:14:40.499 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:14:40.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:40.500 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:14:40.573 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:14:40.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:40.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:14:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:40.693 INFO:teuthology.orchestra.run.smithi164.stdout:client5019 2024-07-26T06:14:40.693 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:40.693 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:40.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-07-26T06:14:40.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.956+0000 7f4f9e200640 1 -- 172.21.15.163:0/4141187670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f78005e50 msgr2=0x7f4f780062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.956+0000 7f4f9e200640 1 --2- 172.21.15.163:0/4141187670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f78005e50 0x7f4f780062d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4f98068e30 tx=0x7f4f78006940 comp rx=0 tx=0).stop 2024-07-26T06:14:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.956+0000 7f4f9d800640 1 -- 172.21.15.163:0/4141187670 reap_dead start 2024-07-26T06:14:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.956+0000 7f4f9e200640 1 -- 172.21.15.163:0/4141187670 shutdown_connections 2024-07-26T06:14:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.957+0000 7f4f9e200640 1 -- 172.21.15.163:0/4141187670 >> 172.21.15.163:0/4141187670 conn(0x7f4f980b0e30 msgr2=0x7f4f980b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.957+0000 7f4f9e200640 1 -- 172.21.15.163:0/4141187670 shutdown_connections 2024-07-26T06:14:40.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.957+0000 7f4f9e200640 1 -- 172.21.15.163:0/4141187670 wait complete. 2024-07-26T06:14:40.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.958+0000 7f4f9e200640 1 Processor -- start 2024-07-26T06:14:40.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.958+0000 7f4f9e200640 1 -- start start 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.959+0000 7f4f9e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f78005e50 0x7f4f98073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.959+0000 7f4f9e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f981e7460 0x7f4f981e7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.959+0000 7f4f9e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 0x7f4f981e8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.959+0000 7f4f9e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980b2d80 con 0x7f4f981e81c0 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.959+0000 7f4f9e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9806f500 con 0x7f4f78005e50 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.959+0000 7f4f9e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980b2f00 con 0x7f4f981e7460 2024-07-26T06:14:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 0x7f4f981e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 0x7f4f981e8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40634/0 (socket says 172.21.15.163:40634) 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 -- 172.21.15.163:0/3517637587 learned_addr learned my addr 172.21.15.163:0/3517637587 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f8fe00640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f981e7460 0x7f4f981e7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9ce00640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f78005e50 0x7f4f98073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 -- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f981e7460 msgr2=0x7f4f981e7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f981e7460 0x7f4f981e7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 -- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f78005e50 msgr2=0x7f4f98073290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f78005e50 0x7f4f98073290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 -- 172.21.15.163:0/3517637587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f98073af0 con 0x7f4f981e81c0 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.960+0000 7f4f9d800640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 0x7f4f981e8670 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f90001e80 tx=0x7f4f90006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.961+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f900062b0 con 0x7f4f981e81c0 2024-07-26T06:14:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.961+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f90006450 con 0x7f4f981e81c0 2024-07-26T06:14:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.961+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90006880 con 0x7f4f981e81c0 2024-07-26T06:14:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.961+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f98073dd0 con 0x7f4f981e81c0 2024-07-26T06:14:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.961+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f981e3140 con 0x7f4f981e81c0 2024-07-26T06:14:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.962+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f980cd530 con 0x7f4f981e81c0 2024-07-26T06:14:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.963+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4f90003a30 con 0x7f4f981e81c0 2024-07-26T06:14:40.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.965+0000 7f4f95600640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f74076300 0x7f4f740787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:40.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.965+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f4f900b7460 con 0x7f4f981e81c0 2024-07-26T06:14:40.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.965+0000 7f4f9ce00640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f74076300 0x7f4f740787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:40.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.966+0000 7f4f9ce00640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f74076300 0x7f4f740787c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001eb0 tx=0x7f4f80001d10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:40.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:40.969+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f90080db0 con 0x7f4f981e81c0 2024-07-26T06:14:41.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.221+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f4f980737d0 con 0x7f4f981e81c0 2024-07-26T06:14:41.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.990+0000 7f4f95600640 1 -- 172.21.15.163:0/3517637587 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) ==== 136+0+0 (secure 0 0 0) 0x7f4f900806b0 con 0x7f4f981e81c0 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.993+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f74076300 msgr2=0x7f4f740787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.993+0000 7f4f9e200640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f74076300 0x7f4f740787c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001eb0 tx=0x7f4f80001d10 comp rx=0 tx=0).stop 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 msgr2=0x7f4f981e8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 0x7f4f981e8670 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f90001e80 tx=0x7f4f90006ca0 comp rx=0 tx=0).stop 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 shutdown_connections 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f74076300 0x7f4f740787c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f981e81c0 0x7f4f981e8670 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:41.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f981e7460 0x7f4f981e7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:41.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 --2- 172.21.15.163:0/3517637587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f78005e50 0x7f4f98073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:41.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 >> 172.21.15.163:0/3517637587 conn(0x7f4f980b0e30 msgr2=0x7f4f980afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:41.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 shutdown_connections 2024-07-26T06:14:41.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:41.994+0000 7f4f9e200640 1 -- 172.21.15.163:0/3517637587 wait complete. 2024-07-26T06:14:42.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 -- 172.21.15.163:0/1258464111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe48406bf40 msgr2=0x7fe48406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 --2- 172.21.15.163:0/1258464111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe48406bf40 0x7fe48406c360 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe4840aec80 tx=0x7fe47002a560 comp rx=0 tx=0).stop 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 -- 172.21.15.163:0/1258464111 shutdown_connections 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 --2- 172.21.15.163:0/1258464111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4841429d0 0x7fe484142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 --2- 172.21.15.163:0/1258464111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe48406bf40 0x7fe48406c360 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 --2- 172.21.15.163:0/1258464111 >> v2:172.21.15.164:3300/0 conn(0x7fe4841404a0 0x7fe484161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 -- 172.21.15.163:0/1258464111 >> 172.21.15.163:0/1258464111 conn(0x7fe48406d420 msgr2=0x7fe48406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 -- 172.21.15.163:0/1258464111 shutdown_connections 2024-07-26T06:14:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.225+0000 7fe48ba00640 1 -- 172.21.15.163:0/1258464111 wait complete. 2024-07-26T06:14:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.227+0000 7fe48ba00640 1 Processor -- start 2024-07-26T06:14:42.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.227+0000 7fe48ba00640 1 -- start start 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe48ba00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe48406bf40 0x7fe4841f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe48ba00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4841404a0 0x7fe4840b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe48ba00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 0x7fe48415bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe48ba00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe484161250 con 0x7fe4841429d0 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe48ba00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe484161ec0 con 0x7fe48406bf40 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe48ba00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe484143380 con 0x7fe4841404a0 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 0x7fe48415bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 0x7fe48415bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40652/0 (socket says 172.21.15.163:40652) 2024-07-26T06:14:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 -- 172.21.15.163:0/563914524 learned_addr learned my addr 172.21.15.163:0/563914524 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489000640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe48406bf40 0x7fe4841f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe483e00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4841404a0 0x7fe4840b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 -- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4841404a0 msgr2=0x7fe4840b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4841404a0 0x7fe4840b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 -- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe48406bf40 msgr2=0x7fe4841f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe48406bf40 0x7fe4841f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.228+0000 7fe489a00640 1 -- 172.21.15.163:0/563914524 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4840b1600 con 0x7fe4841429d0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.229+0000 7fe489a00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 0x7fe48415bc20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00cab0 tx=0x7fe47c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.229+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c007d90 con 0x7fe4841429d0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.229+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe47c00f540 con 0x7fe4841429d0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.229+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00f990 con 0x7fe4841429d0 2024-07-26T06:14:42.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.229+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4840c43c0 con 0x7fe4841429d0 2024-07-26T06:14:42.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.229+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4840b12c0 con 0x7fe4841429d0 2024-07-26T06:14:42.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.230+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe484143890 con 0x7fe4841429d0 2024-07-26T06:14:42.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.232+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe47c00fb30 con 0x7fe4841429d0 2024-07-26T06:14:42.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.232+0000 7fe480c00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4640762d0 0x7fe464078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.233+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fe47c098100 con 0x7fe4841429d0 2024-07-26T06:14:42.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.233+0000 7fe489000640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4640762d0 0x7fe464078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.234+0000 7fe489000640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4640762d0 0x7fe464078790 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe478005fd0 tx=0x7fe478005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:42.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.237+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe47c014070 con 0x7fe4841429d0 2024-07-26T06:14:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.468+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fe48406cd80 con 0x7fe4640762d0 2024-07-26T06:14:42.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.472+0000 7fe480c00640 1 -- 172.21.15.163:0/563914524 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fe48406cd80 con 0x7fe4640762d0 2024-07-26T06:14:42.472 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:14:42.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:14:42.472 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephfs" 2024-07-26T06:14:42.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:14:42.472 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:14:42.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4640762d0 msgr2=0x7fe464078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4640762d0 0x7fe464078790 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe478005fd0 tx=0x7fe478005850 comp rx=0 tx=0).stop 2024-07-26T06:14:42.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 msgr2=0x7fe48415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 0x7fe48415bc20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00cab0 tx=0x7fe47c00cf70 comp rx=0 tx=0).stop 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 shutdown_connections 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4640762d0 0x7fe464078790 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4841429d0 0x7fe48415bc20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4841404a0 0x7fe4840b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 --2- 172.21.15.163:0/563914524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe48406bf40 0x7fe4841f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 >> 172.21.15.163:0/563914524 conn(0x7fe48406d420 msgr2=0x7fe4840732f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.474+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 shutdown_connections 2024-07-26T06:14:42.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.475+0000 7fe48ba00640 1 -- 172.21.15.163:0/563914524 wait complete. 2024-07-26T06:14:42.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:14:42.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 -- 172.21.15.163:0/358659961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 msgr2=0x7f8e040ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 --2- 172.21.15.163:0/358659961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e040ad8e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8e0415e970 tx=0x7f8df402a560 comp rx=0 tx=0).stop 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 -- 172.21.15.163:0/358659961 shutdown_connections 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 --2- 172.21.15.163:0/358659961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 0x7f8e040cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 --2- 172.21.15.163:0/358659961 >> v2:172.21.15.164:3300/0 conn(0x7f8e040ade20 0x7f8e040cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 --2- 172.21.15.163:0/358659961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e040ad8e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 -- 172.21.15.163:0/358659961 >> 172.21.15.163:0/358659961 conn(0x7f8e040b0e30 msgr2=0x7f8e040b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 -- 172.21.15.163:0/358659961 shutdown_connections 2024-07-26T06:14:42.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.708+0000 7f8e09c00640 1 -- 172.21.15.163:0/358659961 wait complete. 2024-07-26T06:14:42.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.710+0000 7f8e09c00640 1 Processor -- start 2024-07-26T06:14:42.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.711+0000 7f8e09c00640 1 -- start start 2024-07-26T06:14:42.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e040ade20 0x7f8e0413bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 0x7f8e0413a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e04073820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e040ce4d0 con 0x7f8e04146be0 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8e040ce350 con 0x7f8e040ade20 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8e040ce650 con 0x7f8e040cc280 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e04073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e04073820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40680/0 (socket says 172.21.15.163:40680) 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09200640 1 -- 172.21.15.163:0/127902241 learned_addr learned my addr 172.21.15.163:0/127902241 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e03e00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e040ade20 0x7f8e0413bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8dfb400640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 0x7f8e0413a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.712+0000 7f8e09200640 1 -- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 msgr2=0x7f8e0413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09200640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 0x7f8e0413a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09200640 1 -- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e040ade20 msgr2=0x7f8e0413bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09200640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e040ade20 0x7f8e0413bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09200640 1 -- 172.21.15.163:0/127902241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e04074190 con 0x7f8e04146be0 2024-07-26T06:14:42.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8dfb400640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 0x7f8e0413a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:42.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e03e00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e040ade20 0x7f8e0413bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09200640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e04073820 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00b970 tx=0x7f8dfc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc019070 con 0x7f8e04146be0 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dfc00cc50 con 0x7f8e04146be0 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0146a0 con 0x7f8e04146be0 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e040713d0 con 0x7f8e04146be0 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.713+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e04071990 con 0x7f8e04146be0 2024-07-26T06:14:42.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.715+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e04071b90 con 0x7f8e04146be0 2024-07-26T06:14:42.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.716+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8dfc01c070 con 0x7f8e04146be0 2024-07-26T06:14:42.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.717+0000 7f8e00c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8de00762d0 0x7f8de0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:42.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.717+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f8dfc0975a0 con 0x7f8e04146be0 2024-07-26T06:14:42.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.717+0000 7f8e03e00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8de00762d0 0x7f8de0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:42.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.719+0000 7f8e03e00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8de00762d0 0x7f8de0078790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8df4033000 tx=0x7f8df4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:42.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.721+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8e04071b90 con 0x7f8e04146be0 2024-07-26T06:14:42.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.929+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8e040ae8b0 con 0x7f8e04146be0 2024-07-26T06:14:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.930+0000 7f8e00c00640 1 -- 172.21.15.163:0/127902241 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f8dfc060ef0 con 0x7f8e04146be0 2024-07-26T06:14:42.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8de00762d0 msgr2=0x7f8de0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8de00762d0 0x7f8de0078790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8df4033000 tx=0x7f8df4004730 comp rx=0 tx=0).stop 2024-07-26T06:14:42.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 msgr2=0x7f8e04073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:42.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e04073820 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00b970 tx=0x7f8dfc00be30 comp rx=0 tx=0).stop 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 shutdown_connections 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8de00762d0 0x7f8de0078790 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e04146be0 0x7f8e04073820 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e040cc280 0x7f8e0413a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 --2- 172.21.15.163:0/127902241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e040ade20 0x7f8e0413bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 >> 172.21.15.163:0/127902241 conn(0x7f8e040b0e30 msgr2=0x7f8e04160280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 shutdown_connections 2024-07-26T06:14:42.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:42.933+0000 7f8e09c00640 1 -- 172.21.15.163:0/127902241 wait complete. 2024-07-26T06:14:42.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:42.954 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp6130/subvol13380) 2024-07-26T06:14:43.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:43.047 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-07-26T06:14:43.103 INFO:teuthology.orchestra.run.smithi163.stdout:cephfs_data 2024-07-26T06:14:43.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:43.103 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp6130/subvol13380) 2024-07-26T06:14:43.204 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:43.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp6130/subvol13380/.meta 2024-07-26T06:14:43.295 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol13380 subvol_grp6130 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.500+0000 7f332b200640 1 -- 172.21.15.163:0/1711026732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0adff0 msgr2=0x7f332c0cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.500+0000 7f332b200640 1 --2- 172.21.15.163:0/1711026732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0adff0 0x7f332c0cbd40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3314009830 tx=0x7f331402a510 comp rx=0 tx=0).stop 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.500+0000 7f332b200640 1 -- 172.21.15.163:0/1711026732 shutdown_connections 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.500+0000 7f332b200640 1 --2- 172.21.15.163:0/1711026732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f332c0cc280 0x7f332c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.500+0000 7f332b200640 1 --2- 172.21.15.163:0/1711026732 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0adff0 0x7f332c0cbd40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.500+0000 7f332b200640 1 -- 172.21.15.163:0/1711026732 >> 172.21.15.163:0/1711026732 conn(0x7f332c0b0e30 msgr2=0x7f332c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.501+0000 7f332b200640 1 -- 172.21.15.163:0/1711026732 shutdown_connections 2024-07-26T06:14:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.501+0000 7f332b200640 1 -- 172.21.15.163:0/1711026732 wait complete. 2024-07-26T06:14:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.502+0000 7f332b200640 1 Processor -- start 2024-07-26T06:14:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.502+0000 7f332b200640 1 -- start start 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.503+0000 7f332b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f332c0adff0 0x7f332c15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.503+0000 7f332b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 0x7f332c1f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.503+0000 7f332b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f332c146be0 0x7f332c1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.503+0000 7f332b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f332c0b4020 con 0x7f332c0cc280 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.503+0000 7f332b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f332c0b3ea0 con 0x7f332c146be0 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.503+0000 7f332b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f332c0b41a0 con 0x7f332c0adff0 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 0x7f332c1f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 0x7f332c1f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40688/0 (socket says 172.21.15.163:40688) 2024-07-26T06:14:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 -- 172.21.15.163:0/2923948126 learned_addr learned my addr 172.21.15.163:0/2923948126 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3329e00640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f332c0adff0 0x7f332c15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f332a800640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f332c146be0 0x7f332c1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 -- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f332c0adff0 msgr2=0x7f332c15cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f332c0adff0 0x7f332c15cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 -- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f332c146be0 msgr2=0x7f332c1448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f332c146be0 0x7f332c1448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f3323e00640 1 -- 172.21.15.163:0/2923948126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c073930 con 0x7f332c0cc280 2024-07-26T06:14:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.504+0000 7f332a800640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f332c146be0 0x7f332c1448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:43.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.505+0000 7f3323e00640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 0x7f332c1f4260 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3314009950 tx=0x7f3314032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.505+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314040070 con 0x7f332c0cc280 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.505+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33140038e0 con 0x7f332c0cc280 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.505+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314003d30 con 0x7f332c0cc280 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.505+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c144300 con 0x7f332c0cc280 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.505+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c1445e0 con 0x7f332c0cc280 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.506+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332c06a1f0 con 0x7f332c0cc280 2024-07-26T06:14:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.507+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3314039040 con 0x7f332c0cc280 2024-07-26T06:14:43.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.508+0000 7f3322000640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f33080762d0 0x7f3308078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:43.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.509+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f33140b8f40 con 0x7f332c0cc280 2024-07-26T06:14:43.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.509+0000 7f3329e00640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f33080762d0 0x7f3308078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:43.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.510+0000 7f3329e00640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f33080762d0 0x7f3308078790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f332c0b2f20 tx=0x7f331c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:43.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.513+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f332c06a1f0 con 0x7f332c0cc280 2024-07-26T06:14:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.752+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol13380", "group_name": "subvol_grp6130", "target": ["mon-mgr", ""]}) -- 0x7f332c0b4700 con 0x7f33080762d0 2024-07-26T06:14:43.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:43.764+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:43.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:43.764+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:43.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:43.764+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:43.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:43.764+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:43.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:43.764+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:43.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.855+0000 7f3322000640 1 -- 172.21.15.163:0/2923948126 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7f332c0b4700 con 0x7f33080762d0 2024-07-26T06:14:43.855 INFO:teuthology.orchestra.run.smithi163.stdout:/volumes/subvol_grp6130/subvol13380 2024-07-26T06:14:43.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f33080762d0 msgr2=0x7f3308078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f33080762d0 0x7f3308078790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f332c0b2f20 tx=0x7f331c009040 comp rx=0 tx=0).stop 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 msgr2=0x7f332c1f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 0x7f332c1f4260 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3314009950 tx=0x7f3314032c10 comp rx=0 tx=0).stop 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 shutdown_connections 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f33080762d0 0x7f3308078790 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f332c146be0 0x7f332c1448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f332c0cc280 0x7f332c1f4260 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 --2- 172.21.15.163:0/2923948126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f332c0adff0 0x7f332c15cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 >> 172.21.15.163:0/2923948126 conn(0x7f332c0b0e30 msgr2=0x7f332c0b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 shutdown_connections 2024-07-26T06:14:43.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:43.858+0000 7f332b200640 1 -- 172.21.15.163:0/2923948126 wait complete. 2024-07-26T06:14:43.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol13380 subvol_grp6130 2024-07-26T06:14:44.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 -- 172.21.15.163:0/3397640214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a9415d5a0 msgr2=0x7f7a940ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:44.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 --2- 172.21.15.163:0/3397640214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a9415d5a0 0x7f7a940ce2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a84005fd0 tx=0x7f7a8402a4b0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 -- 172.21.15.163:0/3397640214 shutdown_connections 2024-07-26T06:14:44.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 --2- 172.21.15.163:0/3397640214 >> v2:172.21.15.164:3300/0 conn(0x7f7a940b1e80 0x7f7a940b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 --2- 172.21.15.163:0/3397640214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a9415d5a0 0x7f7a940ce2f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 -- 172.21.15.163:0/3397640214 >> 172.21.15.163:0/3397640214 conn(0x7f7a940cfda0 msgr2=0x7f7a940d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:44.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 -- 172.21.15.163:0/3397640214 shutdown_connections 2024-07-26T06:14:44.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.083+0000 7f7a9a800640 1 -- 172.21.15.163:0/3397640214 wait complete. 2024-07-26T06:14:44.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.085+0000 7f7a9a800640 1 Processor -- start 2024-07-26T06:14:44.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.085+0000 7f7a9a800640 1 -- start start 2024-07-26T06:14:44.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.086+0000 7f7a9a800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a940b1e80 0x7f7a940c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:44.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.086+0000 7f7a9a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 0x7f7a9414c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.086+0000 7f7a9a800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a9415d5a0 0x7f7a9415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.086+0000 7f7a9a800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a940b5130 con 0x7f7a940ce830 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.086+0000 7f7a9a800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7a940b4fb0 con 0x7f7a9415d5a0 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.086+0000 7f7a9a800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7a940b52b0 con 0x7f7a940b1e80 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 0x7f7a9414c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 0x7f7a9414c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40704/0 (socket says 172.21.15.163:40704) 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 -- 172.21.15.163:0/2867869238 learned_addr learned my addr 172.21.15.163:0/2867869238 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a99400640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a940b1e80 0x7f7a940c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a99e00640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a9415d5a0 0x7f7a9415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 -- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a940b1e80 msgr2=0x7f7a940c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:44.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a940b1e80 0x7f7a940c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 -- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a9415d5a0 msgr2=0x7f7a9415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a9415d5a0 0x7f7a9415d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.087+0000 7f7a98a00640 1 -- 172.21.15.163:0/2867869238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a94151bf0 con 0x7f7a940ce830 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.088+0000 7f7a98a00640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 0x7f7a9414c950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c790 tx=0x7f7a8000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.088+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80003b70 con 0x7f7a940ce830 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.088+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a80003d10 con 0x7f7a940ce830 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.088+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8000aab0 con 0x7f7a940ce830 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.088+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a94151ed0 con 0x7f7a940ce830 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.088+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a94152490 con 0x7f7a940ce830 2024-07-26T06:14:44.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.089+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a940cefc0 con 0x7f7a940ce830 2024-07-26T06:14:44.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.090+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a8000ac50 con 0x7f7a940ce830 2024-07-26T06:14:44.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.092+0000 7f7a91600640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7a740762d0 0x7f7a74078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:44.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.092+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f7a80097770 con 0x7f7a940ce830 2024-07-26T06:14:44.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.093+0000 7f7a99400640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7a740762d0 0x7f7a74078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:44.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.094+0000 7f7a99400640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7a740762d0 0x7f7a74078790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a84033000 tx=0x7f7a84004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:44.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.096+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a940cefc0 con 0x7f7a940ce830 2024-07-26T06:14:44.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.318+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol13380", "group_name": "subvol_grp6130", "target": ["mon-mgr", ""]}) -- 0x7f7a940b4950 con 0x7f7a740762d0 2024-07-26T06:14:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.330+0000 7f7a91600640 1 -- 172.21.15.163:0/2867869238 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7f7a940b4950 con 0x7f7a740762d0 2024-07-26T06:14:44.330 INFO:teuthology.orchestra.run.smithi163.stdout:/volumes/subvol_grp6130/subvol13380 2024-07-26T06:14:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7a740762d0 msgr2=0x7f7a74078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7a740762d0 0x7f7a74078790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a84033000 tx=0x7f7a84004730 comp rx=0 tx=0).stop 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 msgr2=0x7f7a9414c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 0x7f7a9414c950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000c790 tx=0x7f7a8000cc50 comp rx=0 tx=0).stop 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 shutdown_connections 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7a740762d0 0x7f7a74078790 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a9415d5a0 0x7f7a9415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a940ce830 0x7f7a9414c950 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 --2- 172.21.15.163:0/2867869238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a940b1e80 0x7f7a940c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.332+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 >> 172.21.15.163:0/2867869238 conn(0x7f7a940cfda0 msgr2=0x7f7a9415ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.333+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 shutdown_connections 2024-07-26T06:14:44.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:44.333+0000 7f7a9a800640 1 -- 172.21.15.163:0/2867869238 wait complete. 2024-07-26T06:14:44.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:44.352 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/ad3dbdb2c5b1bfdb3820eff0d16ca6fb.meta) 2024-07-26T06:14:48.075 INFO:teuthology.orchestra.run.smithi163.stdout:[GLOBAL] 2024-07-26T06:14:48.075 INFO:teuthology.orchestra.run.smithi163.stdout:version = 1 2024-07-26T06:14:48.076 INFO:teuthology.orchestra.run.smithi163.stdout:type = subvolume 2024-07-26T06:14:48.076 INFO:teuthology.orchestra.run.smithi163.stdout:path = /volumes/subvol_grp6130/subvol13380 2024-07-26T06:14:48.076 INFO:teuthology.orchestra.run.smithi163.stdout:state = complete 2024-07-26T06:14:48.076 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:48.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol13380 subvol_grp6130 2024-07-26T06:14:48.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 -- 172.21.15.163:0/1810991080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f01413b0 msgr2=0x7fc7f015b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 --2- 172.21.15.163:0/1810991080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f01413b0 0x7fc7f015b610 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc005fd0 tx=0x7fc7dc02a5d0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 -- 172.21.15.163:0/1810991080 shutdown_connections 2024-07-26T06:14:48.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 --2- 172.21.15.163:0/1810991080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f01413b0 0x7fc7f015b610 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 --2- 172.21.15.163:0/1810991080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7f00cdd90 0x7fc7f00ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 -- 172.21.15.163:0/1810991080 >> 172.21.15.163:0/1810991080 conn(0x7fc7f006bf40 msgr2=0x7fc7f006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:48.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 -- 172.21.15.163:0/1810991080 shutdown_connections 2024-07-26T06:14:48.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.293+0000 7fc7f6800640 1 -- 172.21.15.163:0/1810991080 wait complete. 2024-07-26T06:14:48.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.295+0000 7fc7f6800640 1 Processor -- start 2024-07-26T06:14:48.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.295+0000 7fc7f6800640 1 -- start start 2024-07-26T06:14:48.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7f6800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7f0074020 0x7fc7f013d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7f6800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 0x7fc7f01580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7f6800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7f01413b0 0x7fc7f01e5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7f6800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f013ef70 con 0x7fc7f00cdd90 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7f6800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f013edf0 con 0x7fc7f01413b0 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7f6800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f00744d0 con 0x7fc7f0074020 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7eea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 0x7fc7f01580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7eea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 0x7fc7f01580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49520/0 (socket says 172.21.15.163:49520) 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.296+0000 7fc7eea00640 1 -- 172.21.15.163:0/452988952 learned_addr learned my addr 172.21.15.163:0/452988952 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7ef400640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7f0074020 0x7fc7f013d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7eea00640 1 -- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7f0074020 msgr2=0x7fc7f013d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7eea00640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7f0074020 0x7fc7f013d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7eea00640 1 -- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7f01413b0 msgr2=0x7fc7f01e5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7eea00640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7f01413b0 0x7fc7f01e5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7eea00640 1 -- 172.21.15.163:0/452988952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f01f2bc0 con 0x7fc7f00cdd90 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.297+0000 7fc7eea00640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 0x7fc7f01580e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc034040 tx=0x7fc7dc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.298+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc03e030 con 0x7fc7f00cdd90 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.298+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7dc003ca0 con 0x7fc7f00cdd90 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.298+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f0157040 con 0x7fc7f00cdd90 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.298+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc035040 con 0x7fc7f00cdd90 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.298+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f0157600 con 0x7fc7f00cdd90 2024-07-26T06:14:48.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.299+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f0158620 con 0x7fc7f00cdd90 2024-07-26T06:14:48.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.300+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 6) ==== 98252+0+0 (secure 0 0 0) 0x7fc7dc041030 con 0x7fc7f00cdd90 2024-07-26T06:14:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.301+0000 7fc7e7400640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7d0076360 0x7fc7d0078820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.301+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc7dc0b7ff0 con 0x7fc7f00cdd90 2024-07-26T06:14:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.302+0000 7fc7ef400640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7d0076360 0x7fc7d0078820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.303+0000 7fc7ef400640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7d0076360 0x7fc7d0078820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0005fd0 tx=0x7fc7e0005e40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:48.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.306+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc7f0158620 con 0x7fc7f00cdd90 2024-07-26T06:14:48.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.550+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol13380", "group_name": "subvol_grp6130", "target": ["mon-mgr", ""]}) -- 0x7fc7f01e68d0 con 0x7fc7d0076360 2024-07-26T06:14:48.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.595+0000 7fc7e7400640 1 -- 172.21.15.163:0/452988952 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc7f01e68d0 con 0x7fc7d0076360 2024-07-26T06:14:48.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.598+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7d0076360 msgr2=0x7fc7d0078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.598+0000 7fc7f6800640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7d0076360 0x7fc7d0078820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0005fd0 tx=0x7fc7e0005e40 comp rx=0 tx=0).stop 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.598+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 msgr2=0x7fc7f01580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.598+0000 7fc7f6800640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 0x7fc7f01580e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc034040 tx=0x7fc7dc0038a0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 shutdown_connections 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7d0076360 0x7fc7d0078820 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7f01413b0 0x7fc7f01e5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7f00cdd90 0x7fc7f01580e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 --2- 172.21.15.163:0/452988952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7f0074020 0x7fc7f013d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 >> 172.21.15.163:0/452988952 conn(0x7fc7f006bf40 msgr2=0x7fc7f015a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 shutdown_connections 2024-07-26T06:14:48.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.599+0000 7fc7f6800640 1 -- 172.21.15.163:0/452988952 wait complete. 2024-07-26T06:14:48.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.606+0000 7f6cf4a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.606+0000 7f6cf4a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.606+0000 7f6cf4a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.606+0000 7f6cf4a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.606+0000 7f6cf4a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.621 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-07-26T06:14:48.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.637+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.637+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.637+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.637+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:14:48.637+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:14:48.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:14:48.731 INFO:teuthology.orchestra.run.smithi163.stdout:2 2024-07-26T06:14:48.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:14:48.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp6130 2024-07-26T06:14:48.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 -- 172.21.15.163:0/2258585789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b23d0 msgr2=0x7f24b40b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 --2- 172.21.15.163:0/2258585789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b23d0 0x7f24b40b6bd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f24ac005fd0 tx=0x7f24ac02a820 comp rx=0 tx=0).stop 2024-07-26T06:14:48.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 -- 172.21.15.163:0/2258585789 shutdown_connections 2024-07-26T06:14:48.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 --2- 172.21.15.163:0/2258585789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b23d0 0x7f24b40b6bd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 -- 172.21.15.163:0/2258585789 >> 172.21.15.163:0/2258585789 conn(0x7f24b40cfda0 msgr2=0x7f24b40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:48.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 -- 172.21.15.163:0/2258585789 shutdown_connections 2024-07-26T06:14:48.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.942+0000 7f24bae00640 1 -- 172.21.15.163:0/2258585789 wait complete. 2024-07-26T06:14:48.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.944+0000 7f24bae00640 1 Processor -- start 2024-07-26T06:14:48.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.944+0000 7f24bae00640 1 -- start start 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24bae00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 0x7f24b4155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24bae00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24b40b23d0 0x7f24b41600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24bae00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24b415d5a0 0x7f24b415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24bae00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24b40b5130 con 0x7f24b40b1e80 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24bae00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f24b40b4fb0 con 0x7f24b40b23d0 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24bae00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f24b40b52b0 con 0x7f24b415d5a0 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24b9a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 0x7f24b4155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24b9a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 0x7f24b4155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49532/0 (socket says 172.21.15.163:49532) 2024-07-26T06:14:48.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.945+0000 7f24b9a00640 1 -- 172.21.15.163:0/3249639002 learned_addr learned my addr 172.21.15.163:0/3249639002 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9000640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24b40b23d0 0x7f24b41600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24ba400640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24b415d5a0 0x7f24b415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9a00640 1 -- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24b415d5a0 msgr2=0x7f24b415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9a00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24b415d5a0 0x7f24b415d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9a00640 1 -- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24b40b23d0 msgr2=0x7f24b41600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9a00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24b40b23d0 0x7f24b41600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9a00640 1 -- 172.21.15.163:0/3249639002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b40ccd90 con 0x7f24b40b1e80 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9000640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24b40b23d0 0x7f24b41600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b9a00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 0x7f24b4155180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c430 tx=0x7f24a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8017070 con 0x7f24b40b1e80 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24a80053c0 con 0x7f24b40b1e80 2024-07-26T06:14:48.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b40cbfd0 con 0x7f24b40b1e80 2024-07-26T06:14:48.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8003850 con 0x7f24b40b1e80 2024-07-26T06:14:48.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.946+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b40cc590 con 0x7f24b40b1e80 2024-07-26T06:14:48.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.947+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b40ced00 con 0x7f24b40b1e80 2024-07-26T06:14:48.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.949+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 6) ==== 98252+0+0 (secure 0 0 0) 0x7f24a801a070 con 0x7f24b40b1e80 2024-07-26T06:14:48.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.950+0000 7f24b1600640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2494076360 0x7f2494078820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:48.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.950+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f24a8096bc0 con 0x7f24b40b1e80 2024-07-26T06:14:48.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.950+0000 7f24b9000640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2494076360 0x7f2494078820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:48.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.951+0000 7f24b9000640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2494076360 0x7f2494078820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24b415f6d0 tx=0x7f24a0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:48.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:48.954+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24b40ced00 con 0x7f24b40b1e80 2024-07-26T06:14:49.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.187+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp6130", "target": ["mon-mgr", ""]}) -- 0x7f24b4160630 con 0x7f2494076360 2024-07-26T06:14:49.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.197+0000 7f24b1600640 1 -- 172.21.15.163:0/3249639002 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f24a0001ca0 con 0x7f2494076360 2024-07-26T06:14:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2494076360 msgr2=0x7f2494078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2494076360 0x7f2494078820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24b415f6d0 tx=0x7f24a0009040 comp rx=0 tx=0).stop 2024-07-26T06:14:49.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 msgr2=0x7f24b4155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 0x7f24b4155180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c430 tx=0x7f24a800c8f0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 shutdown_connections 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2494076360 0x7f2494078820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24b415d5a0 0x7f24b415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24b40b23d0 0x7f24b41600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 --2- 172.21.15.163:0/3249639002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24b40b1e80 0x7f24b4155180 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.199+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 >> 172.21.15.163:0/3249639002 conn(0x7f24b40cfda0 msgr2=0x7f24b415aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.200+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 shutdown_connections 2024-07-26T06:14:49.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.200+0000 7f24bae00640 1 -- 172.21.15.163:0/3249639002 wait complete. 2024-07-26T06:14:49.219 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:14:49.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:14:49.325 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:14:49.437 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:14:49.437 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:14:49.438 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:49.438 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:49.496 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:49.496 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:49.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:14:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 -- 172.21.15.163:0/1732256483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 msgr2=0x7f7cb80b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 --2- 172.21.15.163:0/1732256483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80b6790 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80d2580 tx=0x7f7cb402a940 comp rx=0 tx=0).stop 2024-07-26T06:14:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 -- 172.21.15.163:0/1732256483 shutdown_connections 2024-07-26T06:14:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 --2- 172.21.15.163:0/1732256483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80b6790 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 --2- 172.21.15.163:0/1732256483 >> v2:172.21.15.164:3300/0 conn(0x7f7cb815d540 0x7f7cb80d2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 -- 172.21.15.163:0/1732256483 >> 172.21.15.163:0/1732256483 conn(0x7f7cb80b4fa0 msgr2=0x7f7cb80b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 -- 172.21.15.163:0/1732256483 shutdown_connections 2024-07-26T06:14:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.752+0000 7f7cc0800640 1 -- 172.21.15.163:0/1732256483 wait complete. 2024-07-26T06:14:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.754+0000 7f7cc0800640 1 Processor -- start 2024-07-26T06:14:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.754+0000 7f7cc0800640 1 -- start start 2024-07-26T06:14:49.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cc0800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:49.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cc0800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cb80d2e90 0x7f7cb81f7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:49.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cc0800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cb815d540 0x7f7cb81f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:49.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cc0800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb80b6e50 con 0x7f7cb80d0540 2024-07-26T06:14:49.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cc0800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb80b6cd0 con 0x7f7cb80d2e90 2024-07-26T06:14:49.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cc0800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb80b6fd0 con 0x7f7cb815d540 2024-07-26T06:14:49.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cbde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:49.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cbde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49550/0 (socket says 172.21.15.163:49550) 2024-07-26T06:14:49.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.755+0000 7f7cbde00640 1 -- 172.21.15.163:0/2435237104 learned_addr learned my addr 172.21.15.163:0/2435237104 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbd400640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cb80d2e90 0x7f7cb81f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbde00640 1 -- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cb815d540 msgr2=0x7f7cb81f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbe800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cb815d540 0x7f7cb81f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbde00640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cb815d540 0x7f7cb81f7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbde00640 1 -- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cb80d2e90 msgr2=0x7f7cb81f7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbde00640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cb80d2e90 0x7f7cb81f7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbde00640 1 -- 172.21.15.163:0/2435237104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb80c9a10 con 0x7f7cb80d0540 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.756+0000 7f7cbde00640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80cd170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e790 tx=0x7f7cac00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.757+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac00eea0 con 0x7f7cb80d0540 2024-07-26T06:14:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.757+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cac00cdb0 con 0x7f7cb80d0540 2024-07-26T06:14:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.757+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac010690 con 0x7f7cb80d0540 2024-07-26T06:14:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.757+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb81f4da0 con 0x7f7cb80d0540 2024-07-26T06:14:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.757+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb80c9450 con 0x7f7cb80d0540 2024-07-26T06:14:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.758+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb80b6790 con 0x7f7cb80d0540 2024-07-26T06:14:49.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.759+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 6) ==== 98252+0+0 (secure 0 0 0) 0x7f7cac010830 con 0x7f7cb80d0540 2024-07-26T06:14:49.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.760+0000 7f7cb2000640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c9c076360 0x7f7c9c078820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:49.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.760+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f7cac098330 con 0x7f7cb80d0540 2024-07-26T06:14:49.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.761+0000 7f7cbd400640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c9c076360 0x7f7c9c078820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.762+0000 7f7cbd400640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c9c076360 0x7f7c9c078820 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8007fd0 tx=0x7f7ca8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:49.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.764+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7cac061b20 con 0x7f7cb80d0540 2024-07-26T06:14:49.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.993+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7cb806d040 con 0x7f7cb80d0540 2024-07-26T06:14:49.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.994+0000 7f7cb2000640 1 -- 172.21.15.163:0/2435237104 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7f7cac061420 con 0x7f7cb80d0540 2024-07-26T06:14:49.995 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c9c076360 msgr2=0x7f7c9c078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c9c076360 0x7f7c9c078820 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8007fd0 tx=0x7f7ca8009040 comp rx=0 tx=0).stop 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 msgr2=0x7f7cb80cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80cd170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e790 tx=0x7f7cac00ec50 comp rx=0 tx=0).stop 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 shutdown_connections 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7c9c076360 0x7f7c9c078820 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cb815d540 0x7f7cb81f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cb80d2e90 0x7f7cb81f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 --2- 172.21.15.163:0/2435237104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cb80d0540 0x7f7cb80cd170 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:49.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 >> 172.21.15.163:0/2435237104 conn(0x7f7cb80b4fa0 msgr2=0x7f7cb80b5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:49.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 shutdown_connections 2024-07-26T06:14:49.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:49.997+0000 7f7cc0800640 1 -- 172.21.15.163:0/2435237104 wait complete. 2024-07-26T06:14:50.017 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:14:50.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:50.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-07-26T06:14:50.135 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:14:50.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:50.135 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:14:50.198 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:14:50.198 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:50.198 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:50.238 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:50.239 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:50.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:14:50.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 -- 172.21.15.163:0/782850188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 msgr2=0x7fe4fc0b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:50.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 --2- 172.21.15.163:0/782850188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc0b6c00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4005fd0 tx=0x7fe4f402aa90 comp rx=0 tx=0).stop 2024-07-26T06:14:50.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 -- 172.21.15.163:0/782850188 shutdown_connections 2024-07-26T06:14:50.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 --2- 172.21.15.163:0/782850188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc0b6c00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 --2- 172.21.15.163:0/782850188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4fc0cfe30 0x7fe4fc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 -- 172.21.15.163:0/782850188 >> 172.21.15.163:0/782850188 conn(0x7fe4fc0b4fa0 msgr2=0x7fe4fc0b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:50.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 -- 172.21.15.163:0/782850188 shutdown_connections 2024-07-26T06:14:50.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.500+0000 7fe502200640 1 -- 172.21.15.163:0/782850188 wait complete. 2024-07-26T06:14:50.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.502+0000 7fe502200640 1 Processor -- start 2024-07-26T06:14:50.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.502+0000 7fe502200640 1 -- start start 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe502200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc1f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe502200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4fc0cfe30 0x7fe4fc0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe502200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4fc15d540 0x7fe4fc151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe502200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe4fc15daa0 con 0x7fe4fc0b6790 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe502200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe4fc15d920 con 0x7fe4fc0cfe30 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe502200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe4fc0b7140 con 0x7fe4fc15d540 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe4fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc1f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe4fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc1f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49556/0 (socket says 172.21.15.163:49556) 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe4fb400640 1 -- 172.21.15.163:0/3771315702 learned_addr learned my addr 172.21.15.163:0/3771315702 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:50.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe4faa00640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4fc0cfe30 0x7fe4fc0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.504+0000 7fe4fbe00640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4fc15d540 0x7fe4fc151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4fb400640 1 -- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4fc15d540 msgr2=0x7fe4fc151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4fb400640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4fc15d540 0x7fe4fc151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4fb400640 1 -- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4fc0cfe30 msgr2=0x7fe4fc0c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4fb400640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4fc0cfe30 0x7fe4fc0c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4fb400640 1 -- 172.21.15.163:0/3771315702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4fc1f82e0 con 0x7fe4fc0b6790 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4fb400640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc1f7c00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00b3f0 tx=0x7fe4ec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec015020 con 0x7fe4fc0b6790 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4ec011920 con 0x7fe4fc0b6790 2024-07-26T06:14:50.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.505+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4fc1f85c0 con 0x7fe4fc0b6790 2024-07-26T06:14:50.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.506+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec011d70 con 0x7fe4fc0b6790 2024-07-26T06:14:50.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.506+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4fc0aa130 con 0x7fe4fc0b6790 2024-07-26T06:14:50.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.507+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4fc0b6c00 con 0x7fe4fc0b6790 2024-07-26T06:14:50.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.508+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fe4ec018020 con 0x7fe4fc0b6790 2024-07-26T06:14:50.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.509+0000 7fe4f3400640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4dc076480 0x7fe4dc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:50.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.509+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fe4ec097780 con 0x7fe4fc0b6790 2024-07-26T06:14:50.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.510+0000 7fe4faa00640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4dc076480 0x7fe4dc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:50.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.511+0000 7fe4faa00640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4dc076480 0x7fe4dc078940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0cd2f0 tx=0x7fe4e800a4c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:50.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.513+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe4ec01f030 con 0x7fe4fc0b6790 2024-07-26T06:14:50.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.732+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe4fc0cac10 con 0x7fe4fc0b6790 2024-07-26T06:14:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.732+0000 7fe4f3400640 1 -- 172.21.15.163:0/3771315702 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) ==== 80+0+0 (secure 0 0 0) 0x7fe4ec060e50 con 0x7fe4fc0b6790 2024-07-26T06:14:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4dc076480 msgr2=0x7fe4dc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4dc076480 0x7fe4dc078940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0cd2f0 tx=0x7fe4e800a4c0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 msgr2=0x7fe4fc1f7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc1f7c00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00b3f0 tx=0x7fe4ec00b8b0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 shutdown_connections 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe4dc076480 0x7fe4dc078940 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4fc15d540 0x7fe4fc151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4fc0cfe30 0x7fe4fc0c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 --2- 172.21.15.163:0/3771315702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4fc0b6790 0x7fe4fc1f7c00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 >> 172.21.15.163:0/3771315702 conn(0x7fe4fc0b4fa0 msgr2=0x7fe4fc06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:50.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 shutdown_connections 2024-07-26T06:14:50.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:50.735+0000 7fe502200640 1 -- 172.21.15.163:0/3771315702 wait complete. 2024-07-26T06:14:50.754 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:14:50.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:50.754 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-07-26T06:14:50.857 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:14:50.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:14:50.857 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:14:50.906 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:14:50.906 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:14:50.907 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:50.963 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:14:50.964 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:14:50.964 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:14:50.964 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:14:51.022 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:14:51.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:51.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8433e00640 1 -- 172.21.15.163:0/720790048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 msgr2=0x7f8414004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8433e00640 1 --2- 172.21.15.163:0/720790048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f8414004c80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8418006f80 tx=0x7f8414005830 comp rx=0 tx=0).stop 2024-07-26T06:14:51.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8431e00640 1 -- 172.21.15.163:0/720790048 reap_dead start 2024-07-26T06:14:51.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8433e00640 1 -- 172.21.15.163:0/720790048 shutdown_connections 2024-07-26T06:14:51.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8433e00640 1 -- 172.21.15.163:0/720790048 >> 172.21.15.163:0/720790048 conn(0x7f842c06d420 msgr2=0x7f842c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:51.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8433e00640 1 -- 172.21.15.163:0/720790048 shutdown_connections 2024-07-26T06:14:51.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.234+0000 7f8433e00640 1 -- 172.21.15.163:0/720790048 wait complete. 2024-07-26T06:14:51.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.236+0000 7f8433e00640 1 Processor -- start 2024-07-26T06:14:51.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.236+0000 7f8433e00640 1 -- start start 2024-07-26T06:14:51.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8433e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f842c1f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8433e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f842c0ae810 0x7f842c152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8433e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f842c1f2ed0 0x7f842c14c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8433e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f842c0b31a0 con 0x7f8414004800 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8433e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f842c0cdc30 con 0x7f842c1f2ed0 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8433e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f842c0b3320 con 0x7f842c0ae810 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8431400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f842c1f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8431400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f842c1f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49570/0 (socket says 172.21.15.163:49570) 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.237+0000 7f8431400640 1 -- 172.21.15.163:0/1206049096 learned_addr learned my addr 172.21.15.163:0/1206049096 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8431400640 1 -- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f842c0ae810 msgr2=0x7f842c152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8431400640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f842c0ae810 0x7f842c152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8431400640 1 -- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f842c1f2ed0 msgr2=0x7f842c14c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8431400640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f842c1f2ed0 0x7f842c14c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8431400640 1 -- 172.21.15.163:0/1206049096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c1e6a40 con 0x7f8414004800 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8431400640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f842c1f2760 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8420005fd0 tx=0x7f842000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420017070 con 0x7f8414004800 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8420004350 con 0x7f8414004800 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420009a00 con 0x7f8414004800 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.238+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842c1f2ca0 con 0x7f8414004800 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.239+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f842c1e6ce0 con 0x7f8414004800 2024-07-26T06:14:51.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.240+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842c06c380 con 0x7f8414004800 2024-07-26T06:14:51.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.241+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f8420009ba0 con 0x7f8414004800 2024-07-26T06:14:51.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.241+0000 7f8429600640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f840c0764d0 0x7f840c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.242+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f84200978f0 con 0x7f8414004800 2024-07-26T06:14:51.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.242+0000 7f8430a00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f840c0764d0 0x7f840c078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:51.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.243+0000 7f8430a00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f840c0764d0 0x7f840c078990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8418033000 tx=0x7f842c0cbfd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:51.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.245+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8420020070 con 0x7f8414004800 2024-07-26T06:14:51.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.489+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f842c15b5a0 con 0x7f8414004800 2024-07-26T06:14:51.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.491+0000 7f8429600640 1 -- 172.21.15.163:0/1206049096 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7f8420061030 con 0x7f8414004800 2024-07-26T06:14:51.492 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:51.492 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":17,"btime":"2024-07-26T06:14:41:979000+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:14:30.337611+0000","modified":"2024-07-26T06:14:33.428686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.163:6835/941040757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":941040757},{"type":"v1","addr":"172.21.15.163:6835","nonce":941040757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-07-26T06:14:51.492 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 17 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f840c0764d0 msgr2=0x7f840c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f840c0764d0 0x7f840c078990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8418033000 tx=0x7f842c0cbfd0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 msgr2=0x7f842c1f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f842c1f2760 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8420005fd0 tx=0x7f842000db60 comp rx=0 tx=0).stop 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 shutdown_connections 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f840c0764d0 0x7f840c078990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f842c1f2ed0 0x7f842c14c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f842c0ae810 0x7f842c152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 --2- 172.21.15.163:0/1206049096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8414004800 0x7f842c1f2760 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 >> 172.21.15.163:0/1206049096 conn(0x7f842c06d420 msgr2=0x7f842c0b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:51.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 shutdown_connections 2024-07-26T06:14:51.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.494+0000 7f8433e00640 1 -- 172.21.15.163:0/1206049096 wait complete. 2024-07-26T06:14:51.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:51.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 -- 172.21.15.163:0/1739305004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6900ce2d0 msgr2=0x7fc690071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 --2- 172.21.15.163:0/1739305004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6900ce2d0 0x7fc690071530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc67c009830 tx=0x7fc67c02a520 comp rx=0 tx=0).stop 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 -- 172.21.15.163:0/1739305004 shutdown_connections 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 --2- 172.21.15.163:0/1739305004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6900ce2d0 0x7fc690071530 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 --2- 172.21.15.163:0/1739305004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6901556f0 0x7fc6900cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 -- 172.21.15.163:0/1739305004 >> 172.21.15.163:0/1739305004 conn(0x7fc6900b0e30 msgr2=0x7fc6900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 -- 172.21.15.163:0/1739305004 shutdown_connections 2024-07-26T06:14:51.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.720+0000 7fc694e00640 1 -- 172.21.15.163:0/1739305004 wait complete. 2024-07-26T06:14:51.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.722+0000 7fc694e00640 1 Processor -- start 2024-07-26T06:14:51.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.722+0000 7fc694e00640 1 -- start start 2024-07-26T06:14:51.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc694e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc690071a70 0x7fc690155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc694e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6900ce2d0 0x7fc6901e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc694e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 0x7fc69013f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc694e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc69015f310 con 0x7fc6901556f0 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc694e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc690071ee0 con 0x7fc690071a70 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc694e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc69015f490 con 0x7fc6900ce2d0 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc68fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 0x7fc69013f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc68fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 0x7fc69013f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49588/0 (socket says 172.21.15.163:49588) 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.723+0000 7fc68fe00640 1 -- 172.21.15.163:0/830647402 learned_addr learned my addr 172.21.15.163:0/830647402 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68f400640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc690071a70 0x7fc690155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68fe00640 1 -- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6900ce2d0 msgr2=0x7fc6901e50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68fe00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6900ce2d0 0x7fc6901e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68fe00640 1 -- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc690071a70 msgr2=0x7fc690155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68fe00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc690071a70 0x7fc690155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68fe00640 1 -- 172.21.15.163:0/830647402 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc690158a80 con 0x7fc6901556f0 2024-07-26T06:14:51.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68f400640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc690071a70 0x7fc690155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc68fe00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 0x7fc69013f840 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc68800cbf0 tx=0x7fc68800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.724+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688016020 con 0x7fc6901556f0 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.725+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc68800acf0 con 0x7fc6901556f0 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.725+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6901e5630 con 0x7fc6901556f0 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.725+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688012680 con 0x7fc6901556f0 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.725+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6901e5be0 con 0x7fc6901556f0 2024-07-26T06:14:51.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.726+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc690158c50 con 0x7fc6901556f0 2024-07-26T06:14:51.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.727+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fc688019020 con 0x7fc6901556f0 2024-07-26T06:14:51.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.728+0000 7fc687400640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc670076480 0x7fc670078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:51.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.729+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc688097f40 con 0x7fc6901556f0 2024-07-26T06:14:51.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.729+0000 7fc68f400640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc670076480 0x7fc670078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:51.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.730+0000 7fc68f400640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc670076480 0x7fc670078940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc68000ba60 tx=0x7fc680005cf0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:51.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.733+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc688020030 con 0x7fc6901556f0 2024-07-26T06:14:51.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.963+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc690076290 con 0x7fc6901556f0 2024-07-26T06:14:51.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.964+0000 7fc687400640 1 -- 172.21.15.163:0/830647402 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7fc690076290 con 0x7fc6901556f0 2024-07-26T06:14:51.965 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:51.966 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":17,"btime":"2024-07-26T06:14:41:979000+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:14:30.337611+0000","modified":"2024-07-26T06:14:33.428686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.163:6835/941040757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":941040757},{"type":"v1","addr":"172.21.15.163:6835","nonce":941040757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-07-26T06:14:51.966 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 17 2024-07-26T06:14:51.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.966+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc670076480 msgr2=0x7fc670078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc670076480 0x7fc670078940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc68000ba60 tx=0x7fc680005cf0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 msgr2=0x7fc69013f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 0x7fc69013f840 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc68800cbf0 tx=0x7fc68800a430 comp rx=0 tx=0).stop 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 shutdown_connections 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc670076480 0x7fc670078940 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6901556f0 0x7fc69013f840 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6900ce2d0 0x7fc6901e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 --2- 172.21.15.163:0/830647402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc690071a70 0x7fc690155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 >> 172.21.15.163:0/830647402 conn(0x7fc6900b0e30 msgr2=0x7fc69015d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 shutdown_connections 2024-07-26T06:14:51.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:51.967+0000 7fc694e00640 1 -- 172.21.15.163:0/830647402 wait complete. 2024-07-26T06:14:51.987 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:52.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 -- 172.21.15.163:0/2052757924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2750003a00 msgr2=0x7f2750003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 --2- 172.21.15.163:0/2052757924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2750003a00 0x7f2750003e00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f275c00c2e0 tx=0x7f2750005830 comp rx=0 tx=0).stop 2024-07-26T06:14:52.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 -- 172.21.15.163:0/2052757924 shutdown_connections 2024-07-26T06:14:52.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 --2- 172.21.15.163:0/2052757924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2750003a00 0x7f2750003e00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 -- 172.21.15.163:0/2052757924 >> 172.21.15.163:0/2052757924 conn(0x7f276c0b0e30 msgr2=0x7f276c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:52.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 -- 172.21.15.163:0/2052757924 shutdown_connections 2024-07-26T06:14:52.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.185+0000 7f2771c00640 1 -- 172.21.15.163:0/2052757924 wait complete. 2024-07-26T06:14:52.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.187+0000 7f2771c00640 1 Processor -- start 2024-07-26T06:14:52.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.187+0000 7f2771c00640 1 -- start start 2024-07-26T06:14:52.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2750003a00 0x7f276c14f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f276c1494e0 0x7f276c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 0x7f276c1409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f276c0718c0 con 0x7f276c140570 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f276c071740 con 0x7f276c1494e0 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f276c071fd0 con 0x7f2750003a00 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 0x7f276c1409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 0x7f276c1409f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49608/0 (socket says 172.21.15.163:49608) 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.188+0000 7f2771200640 1 -- 172.21.15.163:0/2712680466 learned_addr learned my addr 172.21.15.163:0/2712680466 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771200640 1 -- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2750003a00 msgr2=0x7f276c14f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771200640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2750003a00 0x7f276c14f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2763e00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f276c1494e0 0x7f276c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771200640 1 -- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f276c1494e0 msgr2=0x7f276c153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771200640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f276c1494e0 0x7f276c153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771200640 1 -- 172.21.15.163:0/2712680466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276c149960 con 0x7f276c140570 2024-07-26T06:14:52.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771200640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 0x7f276c1409f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f276c0b1dd0 tx=0x7f276400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:52.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764012630 con 0x7f276c140570 2024-07-26T06:14:52.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2764018070 con 0x7f276c140570 2024-07-26T06:14:52.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764012a70 con 0x7f276c140570 2024-07-26T06:14:52.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276c149be0 con 0x7f276c140570 2024-07-26T06:14:52.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.189+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276c14a1a0 con 0x7f276c140570 2024-07-26T06:14:52.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.191+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f276c0ce120 con 0x7f276c140570 2024-07-26T06:14:52.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.192+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f276401c410 con 0x7f276c140570 2024-07-26T06:14:52.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.193+0000 7f2768c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27480764b0 0x7f2748078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.193+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f2764096d30 con 0x7f276c140570 2024-07-26T06:14:52.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.193+0000 7f276be00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27480764b0 0x7f2748078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.194+0000 7f276be00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27480764b0 0x7f2748078970 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f275c033000 tx=0x7f275c001a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:52.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.197+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2764021060 con 0x7f276c140570 2024-07-26T06:14:52.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.404+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f276c0b34f0 con 0x7f276c140570 2024-07-26T06:14:52.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.406+0000 7f2768c00640 1 -- 172.21.15.163:0/2712680466 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7f276c149960 con 0x7f276c140570 2024-07-26T06:14:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:52.407 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":34,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:31.905523+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:14:28.059274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-07-26T06:14:29.335706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27480764b0 msgr2=0x7f2748078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27480764b0 0x7f2748078970 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f275c033000 tx=0x7f275c001a30 comp rx=0 tx=0).stop 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 msgr2=0x7f276c1409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 0x7f276c1409f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f276c0b1dd0 tx=0x7f276400d9e0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 shutdown_connections 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27480764b0 0x7f2748078970 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f276c140570 0x7f276c1409f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f276c1494e0 0x7f276c153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 --2- 172.21.15.163:0/2712680466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2750003a00 0x7f276c14f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 >> 172.21.15.163:0/2712680466 conn(0x7f276c0b0e30 msgr2=0x7f276c15b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:52.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 shutdown_connections 2024-07-26T06:14:52.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.408+0000 7f2771c00640 1 -- 172.21.15.163:0/2712680466 wait complete. 2024-07-26T06:14:52.428 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:14:52.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.622+0000 7f2a89a00640 1 -- 172.21.15.163:0/2833001970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840cd000 msgr2=0x7f2a8415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.622+0000 7f2a89a00640 1 --2- 172.21.15.163:0/2833001970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840cd000 0x7f2a8415ce50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2a840b1e30 tx=0x7f2a7c02aa10 comp rx=0 tx=0).stop 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 -- 172.21.15.163:0/2833001970 shutdown_connections 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 --2- 172.21.15.163:0/2833001970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840cd000 0x7f2a8415ce50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 --2- 172.21.15.163:0/2833001970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a840b4420 0x7f2a840cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 --2- 172.21.15.163:0/2833001970 >> v2:172.21.15.164:3300/0 conn(0x7f2a840cbad0 0x7f2a840cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 -- 172.21.15.163:0/2833001970 >> 172.21.15.163:0/2833001970 conn(0x7f2a840b0e30 msgr2=0x7f2a840b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 -- 172.21.15.163:0/2833001970 shutdown_connections 2024-07-26T06:14:52.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.623+0000 7f2a89a00640 1 -- 172.21.15.163:0/2833001970 wait complete. 2024-07-26T06:14:52.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.624+0000 7f2a89a00640 1 Processor -- start 2024-07-26T06:14:52.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.624+0000 7f2a89a00640 1 -- start start 2024-07-26T06:14:52.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.625+0000 7f2a89a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a840b4420 0x7f2a84155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.625+0000 7f2a89a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a840cd000 0x7f2a84166c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.625+0000 7f2a89a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 0x7f2a84165a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.625+0000 7f2a89a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84159700 con 0x7f2a840741d0 2024-07-26T06:14:52.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.625+0000 7f2a89a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84159580 con 0x7f2a840cd000 2024-07-26T06:14:52.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.625+0000 7f2a89a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2a84159880 con 0x7f2a840b4420 2024-07-26T06:14:52.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 0x7f2a84165a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 0x7f2a84165a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49614/0 (socket says 172.21.15.163:49614) 2024-07-26T06:14:52.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 -- 172.21.15.163:0/4071719972 learned_addr learned my addr 172.21.15.163:0/4071719972 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a7b400640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a840cd000 0x7f2a84166c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a83e00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a840b4420 0x7f2a84155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 -- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a840b4420 msgr2=0x7f2a84155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a840b4420 0x7f2a84155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 -- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a840cd000 msgr2=0x7f2a84166c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a840cd000 0x7f2a84166c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 -- 172.21.15.163:0/4071719972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a841f4490 con 0x7f2a840741d0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.626+0000 7f2a89000640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 0x7f2a84165a80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009d60 tx=0x7f2a7c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.627+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c033050 con 0x7f2a840741d0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.627+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a7c004550 con 0x7f2a840741d0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.627+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c038450 con 0x7f2a840741d0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.627+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a841f4770 con 0x7f2a840741d0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.627+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a84165560 con 0x7f2a840741d0 2024-07-26T06:14:52.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.628+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a841f49b0 con 0x7f2a840741d0 2024-07-26T06:14:52.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.629+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f2a7c036070 con 0x7f2a840741d0 2024-07-26T06:14:52.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.630+0000 7f2a80c00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2a60076480 0x7f2a60078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:52.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.630+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f2a7c0b8fe0 con 0x7f2a840741d0 2024-07-26T06:14:52.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.631+0000 7f2a83e00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2a60076480 0x7f2a60078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:52.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.632+0000 7f2a83e00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2a60076480 0x7f2a60078940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2a740058c0 tx=0x7f2a74005c80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:52.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.634+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2a7c082550 con 0x7f2a840741d0 2024-07-26T06:14:52.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.860+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a8415ce50 con 0x7f2a840741d0 2024-07-26T06:14:52.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.862+0000 7f2a80c00640 1 -- 172.21.15.163:0/4071719972 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7f2a7c081e50 con 0x7f2a840741d0 2024-07-26T06:14:52.863 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:52.863 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":17,"btime":"2024-07-26T06:14:41:979000+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:14:30.337611+0000","modified":"2024-07-26T06:14:33.428686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.163:6835/941040757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":941040757},{"type":"v1","addr":"172.21.15.163:6835","nonce":941040757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-07-26T06:14:52.863 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 17 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2a60076480 msgr2=0x7f2a60078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2a60076480 0x7f2a60078940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2a740058c0 tx=0x7f2a74005c80 comp rx=0 tx=0).stop 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 msgr2=0x7f2a84165a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 0x7f2a84165a80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009d60 tx=0x7f2a7c003f10 comp rx=0 tx=0).stop 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 shutdown_connections 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2a60076480 0x7f2a60078940 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a840741d0 0x7f2a84165a80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a840cd000 0x7f2a84166c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 --2- 172.21.15.163:0/4071719972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a840b4420 0x7f2a84155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:52.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 >> 172.21.15.163:0/4071719972 conn(0x7f2a840b0e30 msgr2=0x7f2a840af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:52.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 shutdown_connections 2024-07-26T06:14:52.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:52.865+0000 7f2a89a00640 1 -- 172.21.15.163:0/4071719972 wait complete. 2024-07-26T06:14:52.886 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:14:53.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.091+0000 7f8136a00640 1 -- 172.21.15.163:0/4229224921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8118003a50 msgr2=0x7f8118003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.091+0000 7f8136a00640 1 --2- 172.21.15.163:0/4229224921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8118003a50 0x7f8118003e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f812400baf0 tx=0x7f8118005830 comp rx=0 tx=0).stop 2024-07-26T06:14:53.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.091+0000 7f8136000640 1 -- 172.21.15.163:0/4229224921 reap_dead start 2024-07-26T06:14:53.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.092+0000 7f8136a00640 1 -- 172.21.15.163:0/4229224921 shutdown_connections 2024-07-26T06:14:53.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.092+0000 7f8136a00640 1 -- 172.21.15.163:0/4229224921 >> 172.21.15.163:0/4229224921 conn(0x7f81300b0e30 msgr2=0x7f81300b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:53.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.092+0000 7f8136a00640 1 -- 172.21.15.163:0/4229224921 shutdown_connections 2024-07-26T06:14:53.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.092+0000 7f8136a00640 1 -- 172.21.15.163:0/4229224921 wait complete. 2024-07-26T06:14:53.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.093+0000 7f8136a00640 1 Processor -- start 2024-07-26T06:14:53.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.093+0000 7f8136a00640 1 -- start start 2024-07-26T06:14:53.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.094+0000 7f8136a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8118003a50 0x7f8130143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.094+0000 7f8136a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81300a94d0 0x7f813013f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.094+0000 7f8136a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 0x7f813015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.094+0000 7f8136a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81300b3aa0 con 0x7f81301426b0 2024-07-26T06:14:53.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.094+0000 7f8136a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81300b3920 con 0x7f81300a94d0 2024-07-26T06:14:53.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.094+0000 7f8136a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f81300b3c20 con 0x7f8118003a50 2024-07-26T06:14:53.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 0x7f813015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 0x7f813015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49640/0 (socket says 172.21.15.163:49640) 2024-07-26T06:14:53.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 -- 172.21.15.163:0/2147924068 learned_addr learned my addr 172.21.15.163:0/2147924068 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:53.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8134c00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81300a94d0 0x7f813013f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 -- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8118003a50 msgr2=0x7f8130143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8135600640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8118003a50 0x7f8130143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8118003a50 0x7f8130143130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 -- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81300a94d0 msgr2=0x7f813013f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81300a94d0 0x7f813013f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8136000640 1 -- 172.21.15.163:0/2147924068 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8130142b30 con 0x7f81301426b0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.095+0000 7f8134c00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81300a94d0 0x7f813013f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.096+0000 7f8136000640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 0x7f813015a050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8130072020 tx=0x7f812800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.096+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128007bb0 con 0x7f81301426b0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.096+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8128017070 con 0x7f81301426b0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.096+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812800f6b0 con 0x7f81301426b0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.096+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813013fc40 con 0x7f81301426b0 2024-07-26T06:14:53.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.096+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81301401e0 con 0x7f81301426b0 2024-07-26T06:14:53.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.097+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81300cdd90 con 0x7f81301426b0 2024-07-26T06:14:53.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.098+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f812800f850 con 0x7f81301426b0 2024-07-26T06:14:53.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.099+0000 7f812d600640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8110076480 0x7f8110078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.099+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f81280979b0 con 0x7f81301426b0 2024-07-26T06:14:53.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.100+0000 7f8135600640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8110076480 0x7f8110078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.101+0000 7f8135600640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8110076480 0x7f8110078940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8124036000 tx=0x7f81240046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:53.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.103+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8128021020 con 0x7f81301426b0 2024-07-26T06:14:53.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.349+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81300b34f0 con 0x7f81301426b0 2024-07-26T06:14:53.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.350+0000 7f812d600640 1 -- 172.21.15.163:0/2147924068 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5229 (secure 0 0 0) 0x7f8128060ee0 con 0x7f81301426b0 2024-07-26T06:14:53.351 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:53.351 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":17,"btime":"2024-07-26T06:14:41:979000+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:14:30.337611+0000","modified":"2024-07-26T06:14:33.428686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.163:6835/941040757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":941040757},{"type":"v1","addr":"172.21.15.163:6835","nonce":941040757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-07-26T06:14:53.351 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 17 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.352+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8110076480 msgr2=0x7f8110078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.352+0000 7f8136a00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8110076480 0x7f8110078940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8124036000 tx=0x7f81240046c0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 msgr2=0x7f813015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 0x7f813015a050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8130072020 tx=0x7f812800cd00 comp rx=0 tx=0).stop 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 shutdown_connections 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8110076480 0x7f8110078940 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81301426b0 0x7f813015a050 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81300a94d0 0x7f813013f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 --2- 172.21.15.163:0/2147924068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8118003a50 0x7f8130143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 >> 172.21.15.163:0/2147924068 conn(0x7f81300b0e30 msgr2=0x7f8130071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:53.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 shutdown_connections 2024-07-26T06:14:53.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.353+0000 7f8136a00640 1 -- 172.21.15.163:0/2147924068 wait complete. 2024-07-26T06:14:53.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:53.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 -- 172.21.15.163:0/2864486657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 msgr2=0x7fc8dc15be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 --2- 172.21.15.163:0/2864486657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15be40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc070bc0 tx=0x7fc8c802a410 comp rx=0 tx=0).stop 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 -- 172.21.15.163:0/2864486657 shutdown_connections 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 --2- 172.21.15.163:0/2864486657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15be40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 --2- 172.21.15.163:0/2864486657 >> v2:172.21.15.164:3300/0 conn(0x7fc8dc0cb7e0 0x7fc8dc15ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 -- 172.21.15.163:0/2864486657 >> 172.21.15.163:0/2864486657 conn(0x7fc8dc06d420 msgr2=0x7fc8dc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 -- 172.21.15.163:0/2864486657 shutdown_connections 2024-07-26T06:14:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.582+0000 7fc8e4200640 1 -- 172.21.15.163:0/2864486657 wait complete. 2024-07-26T06:14:53.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.584+0000 7fc8e4200640 1 Processor -- start 2024-07-26T06:14:53.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.584+0000 7fc8e4200640 1 -- start start 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e4200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e4200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8dc0cb7e0 0x7fc8dc150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e4200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8dc15c380 0x7fc8dc14ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e4200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc073530 con 0x7fc8dc071cf0 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e4200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc06c860 con 0x7fc8dc0cb7e0 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e4200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc8dc0736b0 con 0x7fc8dc15c380 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e1800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e1800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49662/0 (socket says 172.21.15.163:49662) 2024-07-26T06:14:53.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.585+0000 7fc8e1800640 1 -- 172.21.15.163:0/3136836021 learned_addr learned my addr 172.21.15.163:0/3136836021 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e0e00640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8dc0cb7e0 0x7fc8dc150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e1800640 1 -- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8dc15c380 msgr2=0x7fc8dc14ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e1800640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8dc15c380 0x7fc8dc14ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e1800640 1 -- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8dc0cb7e0 msgr2=0x7fc8dc150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e1800640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8dc0cb7e0 0x7fc8dc150e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e1800640 1 -- 172.21.15.163:0/3136836021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc1517b0 con 0x7fc8dc071cf0 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e1800640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15a720 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c890 tx=0x7fc8d000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0004220 con 0x7fc8dc071cf0 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8d00079b0 con 0x7fc8dc071cf0 2024-07-26T06:14:53.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0007de0 con 0x7fc8dc071cf0 2024-07-26T06:14:53.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc151a90 con 0x7fc8dc071cf0 2024-07-26T06:14:53.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.586+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc152050 con 0x7fc8dc071cf0 2024-07-26T06:14:53.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.587+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8dc15be40 con 0x7fc8dc071cf0 2024-07-26T06:14:53.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.588+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fc8d0005020 con 0x7fc8dc071cf0 2024-07-26T06:14:53.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.589+0000 7fc8d9600640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc8bc076480 0x7fc8bc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:53.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.590+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fc8d0097ff0 con 0x7fc8dc071cf0 2024-07-26T06:14:53.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.590+0000 7fc8e0e00640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc8bc076480 0x7fc8bc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:53.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.591+0000 7fc8e0e00640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc8bc076480 0x7fc8bc078940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8034000 tx=0x7fc8c8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.594+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc8d0014030 con 0x7fc8dc071cf0 2024-07-26T06:14:53.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.786+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8dc0b3650 con 0x7fc8dc071cf0 2024-07-26T06:14:53.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.788+0000 7fc8d9600640 1 -- 172.21.15.163:0/3136836021 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16693 (secure 0 0 0) 0x7fc8d0061540 con 0x7fc8dc071cf0 2024-07-26T06:14:53.791 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:53.791 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":34,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:31.905523+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:14:28.059274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-07-26T06:14:29.335706+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:53.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc8bc076480 msgr2=0x7fc8bc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc8bc076480 0x7fc8bc078940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8034000 tx=0x7fc8c8004730 comp rx=0 tx=0).stop 2024-07-26T06:14:53.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 msgr2=0x7fc8dc15a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:53.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15a720 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c890 tx=0x7fc8d000cd50 comp rx=0 tx=0).stop 2024-07-26T06:14:53.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 shutdown_connections 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc8bc076480 0x7fc8bc078940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8dc15c380 0x7fc8dc14ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8dc0cb7e0 0x7fc8dc150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 --2- 172.21.15.163:0/3136836021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc8dc071cf0 0x7fc8dc15a720 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.790+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 >> 172.21.15.163:0/3136836021 conn(0x7fc8dc06d420 msgr2=0x7fc8dc0b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.791+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 shutdown_connections 2024-07-26T06:14:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:53.791+0000 7fc8e4200640 1 -- 172.21.15.163:0/3136836021 wait complete. 2024-07-26T06:14:53.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:14:54.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 -- 172.21.15.163:0/3790147142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 msgr2=0x7fe75c0ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 --2- 172.21.15.163:0/3790147142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ce2f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe75c06f160 tx=0x7fe74c02a9a0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 -- 172.21.15.163:0/3790147142 shutdown_connections 2024-07-26T06:14:54.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 --2- 172.21.15.163:0/3790147142 >> v2:172.21.15.164:3300/0 conn(0x7fe75c0b1e80 0x7fe75c0b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 --2- 172.21.15.163:0/3790147142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ce2f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 -- 172.21.15.163:0/3790147142 >> 172.21.15.163:0/3790147142 conn(0x7fe75c0cfda0 msgr2=0x7fe75c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:54.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 -- 172.21.15.163:0/3790147142 shutdown_connections 2024-07-26T06:14:54.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.009+0000 7fe761a00640 1 -- 172.21.15.163:0/3790147142 wait complete. 2024-07-26T06:14:54.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.011+0000 7fe761a00640 1 Processor -- start 2024-07-26T06:14:54.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.011+0000 7fe761a00640 1 -- start start 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe75c0b1e80 0x7fe75c15f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75c0ce830 0x7fe75c15fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0b5130 con 0x7fe75c15d5a0 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0b4fb0 con 0x7fe75c0b1e80 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe75c0b52b0 con 0x7fe75c0ce830 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.012+0000 7fe761000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49678/0 (socket says 172.21.15.163:49678) 2024-07-26T06:14:54.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 -- 172.21.15.163:0/280629702 learned_addr learned my addr 172.21.15.163:0/280629702 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe75be00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe75c0b1e80 0x7fe75c15f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe75b400640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75c0ce830 0x7fe75c15fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 -- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75c0ce830 msgr2=0x7fe75c15fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75c0ce830 0x7fe75c15fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 -- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe75c0b1e80 msgr2=0x7fe75c15f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe75c0b1e80 0x7fe75c15f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 -- 172.21.15.163:0/280629702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75c0cb0e0 con 0x7fe75c15d5a0 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe75be00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe75c0b1e80 0x7fe75c15f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:54.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.013+0000 7fe761000640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ca3f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe754009fd0 tx=0x7fe75400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:54.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.014+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754015020 con 0x7fe75c15d5a0 2024-07-26T06:14:54.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.014+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe754011920 con 0x7fe75c15d5a0 2024-07-26T06:14:54.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.014+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754011d50 con 0x7fe75c15d5a0 2024-07-26T06:14:54.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.014+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75c0cb3c0 con 0x7fe75c15d5a0 2024-07-26T06:14:54.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.014+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75c150a00 con 0x7fe75c15d5a0 2024-07-26T06:14:54.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.015+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe75c0ce2f0 con 0x7fe75c15d5a0 2024-07-26T06:14:54.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.016+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fe754018020 con 0x7fe75c15d5a0 2024-07-26T06:14:54.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.017+0000 7fe753e00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe738076480 0x7fe738078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.017+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fe754097a70 con 0x7fe75c15d5a0 2024-07-26T06:14:54.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.018+0000 7fe75be00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe738076480 0x7fe738078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.019+0000 7fe75be00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe738076480 0x7fe738078940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe74c035000 tx=0x7fe74c004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:54.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.021+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe75401f020 con 0x7fe75c15d5a0 2024-07-26T06:14:54.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.268+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe75c150cc0 con 0x7fe75c15d5a0 2024-07-26T06:14:54.270 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:14:54.269+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:14:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.273+0000 7fe753e00640 1 -- 172.21.15.163:0/280629702 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7fe754061140 con 0x7fe75c15d5a0 2024-07-26T06:14:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:14:54.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe738076480 msgr2=0x7fe738078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe738076480 0x7fe738078940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe74c035000 tx=0x7fe74c004ab0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 msgr2=0x7fe75c0ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ca3f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe754009fd0 tx=0x7fe75400b8e0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 shutdown_connections 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe738076480 0x7fe738078940 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe75c15d5a0 0x7fe75c0ca3f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75c0ce830 0x7fe75c15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 --2- 172.21.15.163:0/280629702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe75c0b1e80 0x7fe75c15f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.276+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 >> 172.21.15.163:0/280629702 conn(0x7fe75c0cfda0 msgr2=0x7fe75c0b89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.277+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 shutdown_connections 2024-07-26T06:14:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.277+0000 7fe761a00640 1 -- 172.21.15.163:0/280629702 wait complete. 2024-07-26T06:14:54.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:14:54.392 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:14:54.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 -- 172.21.15.163:0/3728978337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24281590a0 msgr2=0x7f24280b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 --2- 172.21.15.163:0/3728978337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24281590a0 0x7f24280b4c80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f241800a940 tx=0x7f241802b850 comp rx=0 tx=0).stop 2024-07-26T06:14:54.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 -- 172.21.15.163:0/3728978337 shutdown_connections 2024-07-26T06:14:54.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 --2- 172.21.15.163:0/3728978337 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24281590a0 0x7f24280b4c80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 -- 172.21.15.163:0/3728978337 >> 172.21.15.163:0/3728978337 conn(0x7f24280b0e30 msgr2=0x7f24280b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:54.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 -- 172.21.15.163:0/3728978337 shutdown_connections 2024-07-26T06:14:54.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.482+0000 7f2427e00640 1 -- 172.21.15.163:0/3728978337 wait complete. 2024-07-26T06:14:54.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.484+0000 7f2427e00640 1 Processor -- start 2024-07-26T06:14:54.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.484+0000 7f2427e00640 1 -- start start 2024-07-26T06:14:54.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 0x7f242814ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24280cdab0 0x7f2428157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24281590a0 0x7f24281e2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f242806f470 con 0x7f24280ccdc0 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f242806f2f0 con 0x7f24281590a0 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f24280b2840 con 0x7f24280cdab0 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 0x7f242814ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 0x7f242814ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49720/0 (socket says 172.21.15.163:49720) 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 -- 172.21.15.163:0/1069332919 learned_addr learned my addr 172.21.15.163:0/1069332919 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 -- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24280cdab0 msgr2=0x7f2428157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426000640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24280cdab0 0x7f2428157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24280cdab0 0x7f2428157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2427400640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24281590a0 0x7f24281e2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 -- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24281590a0 msgr2=0x7f24281e2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.485+0000 7f2426a00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24281590a0 0x7f24281e2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:54.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f2426a00640 1 -- 172.21.15.163:0/1069332919 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24281f3a10 con 0x7f24280ccdc0 2024-07-26T06:14:54.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f2426a00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 0x7f242814ae40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2418000c00 tx=0x7f2418003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:54.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418041020 con 0x7f24280ccdc0 2024-07-26T06:14:54.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2418031a50 con 0x7f24280ccdc0 2024-07-26T06:14:54.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418003f40 con 0x7f24280ccdc0 2024-07-26T06:14:54.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24281f3cd0 con 0x7f24280ccdc0 2024-07-26T06:14:54.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.486+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24281f4290 con 0x7f24280ccdc0 2024-07-26T06:14:54.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.487+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24280b2dc0 con 0x7f24280ccdc0 2024-07-26T06:14:54.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.488+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f2418044020 con 0x7f24280ccdc0 2024-07-26T06:14:54.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.489+0000 7f241ea00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2408076480 0x7f2408078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:54.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.490+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5795+0+0 (secure 0 0 0) 0x7f24180b8b40 con 0x7f24280ccdc0 2024-07-26T06:14:54.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.490+0000 7f2426000640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2408076480 0x7f2408078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:54.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.491+0000 7f2426000640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2408076480 0x7f2408078940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f24281e3ac0 tx=0x7f2414009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:54.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.493+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f241804b070 con 0x7f24280ccdc0 2024-07-26T06:14:54.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:54.737+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2428155020 con 0x7f24280ccdc0 2024-07-26T06:14:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.276+0000 7f241ea00640 1 -- 172.21.15.163:0/1069332919 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f24180821c0 con 0x7f24280ccdc0 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.281+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2408076480 msgr2=0x7f2408078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.281+0000 7f2427e00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2408076480 0x7f2408078940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f24281e3ac0 tx=0x7f2414009040 comp rx=0 tx=0).stop 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.281+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 msgr2=0x7f242814ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.281+0000 7f2427e00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 0x7f242814ae40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2418000c00 tx=0x7f2418003d50 comp rx=0 tx=0).stop 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 shutdown_connections 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2408076480 0x7f2408078940 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24281590a0 0x7f24281e2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24280cdab0 0x7f2428157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 --2- 172.21.15.163:0/1069332919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24280ccdc0 0x7f242814ae40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 >> 172.21.15.163:0/1069332919 conn(0x7f24280b0e30 msgr2=0x7f242814bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:55.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 shutdown_connections 2024-07-26T06:14:55.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.282+0000 7f2427e00640 1 -- 172.21.15.163:0/1069332919 wait complete. 2024-07-26T06:14:55.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:14:55.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.516+0000 7ffa3a400640 1 -- 172.21.15.163:0/3407259887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa341556f0 msgr2=0x7ffa340cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.516+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3407259887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa341556f0 0x7ffa340cdd90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffa3406cf00 tx=0x7ffa2802a530 comp rx=0 tx=0).stop 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.516+0000 7ffa3a400640 1 -- 172.21.15.163:0/3407259887 shutdown_connections 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.516+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3407259887 >> v2:172.21.15.164:3300/0 conn(0x7ffa340ce2d0 0x7ffa34071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.516+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3407259887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa341556f0 0x7ffa340cdd90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.516+0000 7ffa3a400640 1 -- 172.21.15.163:0/3407259887 >> 172.21.15.163:0/3407259887 conn(0x7ffa340b0e30 msgr2=0x7ffa340b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.517+0000 7ffa3a400640 1 -- 172.21.15.163:0/3407259887 shutdown_connections 2024-07-26T06:14:55.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.517+0000 7ffa3a400640 1 -- 172.21.15.163:0/3407259887 wait complete. 2024-07-26T06:14:55.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.518+0000 7ffa3a400640 1 Processor -- start 2024-07-26T06:14:55.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.518+0000 7ffa3a400640 1 -- start start 2024-07-26T06:14:55.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.519+0000 7ffa3a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 0x7ffa3414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:55.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.519+0000 7ffa3a400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa340ce2d0 0x7ffa3414d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:55.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.519+0000 7ffa3a400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa341556f0 0x7ffa3415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:55.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.519+0000 7ffa3a400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa34072000 con 0x7ffa34071a70 2024-07-26T06:14:55.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.519+0000 7ffa3a400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffa34071e80 con 0x7ffa340ce2d0 2024-07-26T06:14:55.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.519+0000 7ffa3a400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3415f310 con 0x7ffa341556f0 2024-07-26T06:14:55.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 0x7ffa3414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:55.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 0x7ffa3414cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36894/0 (socket says 172.21.15.163:36894) 2024-07-26T06:14:55.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 -- 172.21.15.163:0/3594898218 learned_addr learned my addr 172.21.15.163:0/3594898218 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:55.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39a00640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa341556f0 0x7ffa3415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:55.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa33e00640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa340ce2d0 0x7ffa3414d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 -- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa341556f0 msgr2=0x7ffa3415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa341556f0 0x7ffa3415a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 -- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa340ce2d0 msgr2=0x7ffa3414d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa340ce2d0 0x7ffa3414d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39000640 1 -- 172.21.15.163:0/3594898218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa341502a0 con 0x7ffa34071a70 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.520+0000 7ffa39a00640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa341556f0 0x7ffa3415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.521+0000 7ffa39000640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 0x7ffa3414cae0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009920 tx=0x7ffa28033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.521+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28031d60 con 0x7ffa34071a70 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.521+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa28033e10 con 0x7ffa34071a70 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.521+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28005260 con 0x7ffa34071a70 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.521+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa34150580 con 0x7ffa34071a70 2024-07-26T06:14:55.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.521+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa34150b40 con 0x7ffa34071a70 2024-07-26T06:14:55.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.522+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3415f850 con 0x7ffa34071a70 2024-07-26T06:14:55.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.523+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7ffa28005400 con 0x7ffa34071a70 2024-07-26T06:14:55.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.524+0000 7ffa30c00640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa14076480 0x7ffa14078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:55.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.524+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5795+0+0 (secure 0 0 0) 0x7ffa280b8d20 con 0x7ffa34071a70 2024-07-26T06:14:55.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.525+0000 7ffa33e00640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa14076480 0x7ffa14078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:55.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.526+0000 7ffa33e00640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa14076480 0x7ffa14078940 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffa20005fd0 tx=0x7ffa20005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:55.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.528+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa2803f030 con 0x7ffa34071a70 2024-07-26T06:14:55.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:55.733+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffa340ced60 con 0x7ffa34071a70 2024-07-26T06:14:56.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:14:56.016 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:14:56.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.288+0000 7ffa30c00640 1 -- 172.21.15.163:0/3594898218 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) ==== 181+0+0 (secure 0 0 0) 0x7ffa28082200 con 0x7ffa34071a70 2024-07-26T06:14:56.289 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:14:56.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.292+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa14076480 msgr2=0x7ffa14078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:56.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.292+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa14076480 0x7ffa14078940 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffa20005fd0 tx=0x7ffa20005850 comp rx=0 tx=0).stop 2024-07-26T06:14:56.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.292+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 msgr2=0x7ffa3414cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:56.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.292+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 0x7ffa3414cae0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009920 tx=0x7ffa28033c20 comp rx=0 tx=0).stop 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 shutdown_connections 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ffa14076480 0x7ffa14078940 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa341556f0 0x7ffa3415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa340ce2d0 0x7ffa3414d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 --2- 172.21.15.163:0/3594898218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa34071a70 0x7ffa3414cae0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 >> 172.21.15.163:0/3594898218 conn(0x7ffa340b0e30 msgr2=0x7ffa340b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:56.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 shutdown_connections 2024-07-26T06:14:56.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.293+0000 7ffa3a400640 1 -- 172.21.15.163:0/3594898218 wait complete. 2024-07-26T06:14:56.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.519+0000 7ff58f200640 1 -- 172.21.15.163:0/4197607948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 msgr2=0x7ff590071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.519+0000 7ff58f200640 1 --2- 172.21.15.163:0/4197607948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff590071530 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff59006ce80 tx=0x7ff57802a560 comp rx=0 tx=0).stop 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 -- 172.21.15.163:0/4197607948 shutdown_connections 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 --2- 172.21.15.163:0/4197607948 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590071a70 0x7ff59015ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 --2- 172.21.15.163:0/4197607948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff590071530 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 --2- 172.21.15.163:0/4197607948 >> v2:172.21.15.164:3300/0 conn(0x7ff5901556f0 0x7ff5900cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 -- 172.21.15.163:0/4197607948 >> 172.21.15.163:0/4197607948 conn(0x7ff5900b0e30 msgr2=0x7ff5900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:56.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 -- 172.21.15.163:0/4197607948 shutdown_connections 2024-07-26T06:14:56.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.520+0000 7ff58f200640 1 -- 172.21.15.163:0/4197607948 wait complete. 2024-07-26T06:14:56.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.521+0000 7ff58f200640 1 Processor -- start 2024-07-26T06:14:56.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.522+0000 7ff58f200640 1 -- start start 2024-07-26T06:14:56.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590071a70 0x7ff59013b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:56.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff59014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:56.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5901556f0 0x7ff590149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff59015f2c0 con 0x7ff5900ce2d0 2024-07-26T06:14:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff59015f140 con 0x7ff5901556f0 2024-07-26T06:14:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff59015f440 con 0x7ff590071a70 2024-07-26T06:14:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff59014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:56.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff59014ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36910/0 (socket says 172.21.15.163:36910) 2024-07-26T06:14:56.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 -- 172.21.15.163:0/2832218191 learned_addr learned my addr 172.21.15.163:0/2832218191 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:56.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff58de00640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590071a70 0x7ff59013b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:56.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 -- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590071a70 msgr2=0x7ff59013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:56.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590071a70 0x7ff59013b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 -- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5901556f0 msgr2=0x7ff590149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:56.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5901556f0 0x7ff590149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:56.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.523+0000 7ff585400640 1 -- 172.21.15.163:0/2832218191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59014d770 con 0x7ff5900ce2d0 2024-07-26T06:14:56.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.524+0000 7ff585400640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff59014ccb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff578009850 tx=0x7ff5780039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:56.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.524+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57803f070 con 0x7ff5900ce2d0 2024-07-26T06:14:56.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.524+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff578004890 con 0x7ff5900ce2d0 2024-07-26T06:14:56.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.524+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578035030 con 0x7ff5900ce2d0 2024-07-26T06:14:56.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.524+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5901f7a70 con 0x7ff5900ce2d0 2024-07-26T06:14:56.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.524+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5901f8030 con 0x7ff5900ce2d0 2024-07-26T06:14:56.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.525+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59015f940 con 0x7ff5900ce2d0 2024-07-26T06:14:56.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.526+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7ff578042070 con 0x7ff5900ce2d0 2024-07-26T06:14:56.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.527+0000 7ff586a00640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff56c076480 0x7ff56c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:56.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.528+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5357+0+0 (secure 0 0 0) 0x7ff5780b8940 con 0x7ff5900ce2d0 2024-07-26T06:14:56.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.528+0000 7ff58de00640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff56c076480 0x7ff56c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:56.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.529+0000 7ff58de00640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff56c076480 0x7ff56c078940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff580005fd0 tx=0x7ff580005bc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:56.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.532+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff578082060 con 0x7ff5900ce2d0 2024-07-26T06:14:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:56.737+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff5900ced60 con 0x7ff5900ce2d0 2024-07-26T06:14:57.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.291+0000 7ff586a00640 1 -- 172.21.15.163:0/2832218191 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) ==== 169+0+0 (secure 0 0 0) 0x7ff578081960 con 0x7ff5900ce2d0 2024-07-26T06:14:57.292 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff56c076480 msgr2=0x7ff56c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff56c076480 0x7ff56c078940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff580005fd0 tx=0x7ff580005bc0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 msgr2=0x7ff59014ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff59014ccb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff578009850 tx=0x7ff5780039e0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 shutdown_connections 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff56c076480 0x7ff56c078940 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5901556f0 0x7ff590149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5900ce2d0 0x7ff59014ccb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 --2- 172.21.15.163:0/2832218191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590071a70 0x7ff59013b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 >> 172.21.15.163:0/2832218191 conn(0x7ff5900b0e30 msgr2=0x7ff59015b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:57.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 shutdown_connections 2024-07-26T06:14:57.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.295+0000 7ff58f200640 1 -- 172.21.15.163:0/2832218191 wait complete. 2024-07-26T06:14:57.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-07-26T06:14:57.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 -- 172.21.15.163:0/1274512738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 msgr2=0x7f29b00b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 --2- 172.21.15.163:0/1274512738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b00b40a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f29b00cbd30 tx=0x7f299c02a4b0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 -- 172.21.15.163:0/1274512738 shutdown_connections 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 --2- 172.21.15.163:0/1274512738 >> v2:172.21.15.164:3300/0 conn(0x7f29b015cef0 0x7f29b015d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 --2- 172.21.15.163:0/1274512738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b00b40a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 --2- 172.21.15.163:0/1274512738 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b006f350 0x7f29b00b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 -- 172.21.15.163:0/1274512738 >> 172.21.15.163:0/1274512738 conn(0x7f29b006d420 msgr2=0x7f29b006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.515+0000 7f29b6000640 1 -- 172.21.15.163:0/1274512738 shutdown_connections 2024-07-26T06:14:57.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.516+0000 7f29b6000640 1 -- 172.21.15.163:0/1274512738 wait complete. 2024-07-26T06:14:57.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.517+0000 7f29b6000640 1 Processor -- start 2024-07-26T06:14:57.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.517+0000 7f29b6000640 1 -- start start 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29b6000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29b006f350 0x7f29b015c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29b6000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b015bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29b6000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b015cef0 0x7f29b00af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29b6000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29b006bf40 con 0x7f29b0159380 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29b6000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f29b015dc10 con 0x7f29b006f350 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29b6000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f29b006c0c0 con 0x7f29b015cef0 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29aea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b015bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29aea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b015bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36920/0 (socket says 172.21.15.163:36920) 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.518+0000 7f29aea00640 1 -- 172.21.15.163:0/1083474308 learned_addr learned my addr 172.21.15.163:0/1083474308 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29af400640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29b006f350 0x7f29b015c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29afe00640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b015cef0 0x7f29b00af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29aea00640 1 -- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b015cef0 msgr2=0x7f29b00af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29aea00640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b015cef0 0x7f29b00af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29aea00640 1 -- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29b006f350 msgr2=0x7f29b015c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29aea00640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29b006f350 0x7f29b015c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29aea00640 1 -- 172.21.15.163:0/1083474308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b014bb40 con 0x7f29b0159380 2024-07-26T06:14:57.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29afe00640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b015cef0 0x7f29b00af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:14:57.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29aea00640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b015bed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f299c009850 tx=0x7f299c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c03f070 con 0x7f29b0159380 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f299c004890 con 0x7f29b0159380 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.519+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c035030 con 0x7f29b0159380 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.520+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b014bde0 con 0x7f29b0159380 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.520+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b014c680 con 0x7f29b0159380 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.521+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b006c5e0 con 0x7f29b0159380 2024-07-26T06:14:57.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.522+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f299c042070 con 0x7f29b0159380 2024-07-26T06:14:57.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.523+0000 7f29a7400640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2990076480 0x7f2990078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.523+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f299c0b87e0 con 0x7f29b0159380 2024-07-26T06:14:57.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.523+0000 7f29af400640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2990076480 0x7f2990078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.524+0000 7f29af400640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2990076480 0x7f2990078940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f29a0005fd0 tx=0x7f29a0005da0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:57.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.527+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f299c082010 con 0x7f29b0159380 2024-07-26T06:14:57.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.736+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f29b0155390 con 0x7f29b0159380 2024-07-26T06:14:57.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.744+0000 7f29a7400640 1 -- 172.21.15.163:0/1083474308 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v29) ==== 71+0+0 (secure 0 0 0) 0x7f299c081910 con 0x7f29b0159380 2024-07-26T06:14:57.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2990076480 msgr2=0x7f2990078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2990076480 0x7f2990078940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f29a0005fd0 tx=0x7f29a0005da0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 msgr2=0x7f29b015bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b015bed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f299c009850 tx=0x7f299c0039e0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 shutdown_connections 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2990076480 0x7f2990078940 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29b015cef0 0x7f29b00af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b0159380 0x7f29b015bed0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 --2- 172.21.15.163:0/1083474308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29b006f350 0x7f29b015c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.747+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 >> 172.21.15.163:0/1083474308 conn(0x7f29b006d420 msgr2=0x7f29b0157630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.748+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 shutdown_connections 2024-07-26T06:14:57.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.748+0000 7f29b6000640 1 -- 172.21.15.163:0/1083474308 wait complete. 2024-07-26T06:14:57.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-07-26T06:14:57.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 -- 172.21.15.163:0/423640049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b23d0 msgr2=0x7fb83c0b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 --2- 172.21.15.163:0/423640049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b23d0 0x7fb83c0b6bd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb834009830 tx=0x7fb83402a4a0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 -- 172.21.15.163:0/423640049 shutdown_connections 2024-07-26T06:14:57.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 --2- 172.21.15.163:0/423640049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b23d0 0x7fb83c0b6bd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 -- 172.21.15.163:0/423640049 >> 172.21.15.163:0/423640049 conn(0x7fb83c0cfda0 msgr2=0x7fb83c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:57.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 -- 172.21.15.163:0/423640049 shutdown_connections 2024-07-26T06:14:57.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.972+0000 7fb841c00640 1 -- 172.21.15.163:0/423640049 wait complete. 2024-07-26T06:14:57.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.974+0000 7fb841c00640 1 Processor -- start 2024-07-26T06:14:57.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.974+0000 7fb841c00640 1 -- start start 2024-07-26T06:14:57.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb841c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 0x7fb83c1f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb841c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0b23d0 0x7fb83c15df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb841c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c15d5a0 0x7fb83c145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb841c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb83c0b5130 con 0x7fb83c0b1e80 2024-07-26T06:14:57.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb841c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb83c0b4fb0 con 0x7fb83c0b23d0 2024-07-26T06:14:57.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb841c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb83c0b52b0 con 0x7fb83c15d5a0 2024-07-26T06:14:57.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb83be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 0x7fb83c1f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb83be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 0x7fb83c1f7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36936/0 (socket says 172.21.15.163:36936) 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.975+0000 7fb83be00640 1 -- 172.21.15.163:0/2594491006 learned_addr learned my addr 172.21.15.163:0/2594491006 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83b400640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0b23d0 0x7fb83c15df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb841200640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c15d5a0 0x7fb83c145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83be00640 1 -- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c15d5a0 msgr2=0x7fb83c145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83be00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c15d5a0 0x7fb83c145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83be00640 1 -- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0b23d0 msgr2=0x7fb83c15df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83be00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0b23d0 0x7fb83c15df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:57.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83be00640 1 -- 172.21.15.163:0/2594491006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83c1f7ff0 con 0x7fb83c0b1e80 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.976+0000 7fb83be00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 0x7fb83c1f7ab0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb82400c3a0 tx=0x7fb82400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.977+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824017070 con 0x7fb83c0b1e80 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.977+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8240053c0 con 0x7fb83c0b1e80 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.977+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824003850 con 0x7fb83c0b1e80 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.977+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83c145fe0 con 0x7fb83c0b1e80 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.977+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb83c1f72b0 con 0x7fb83c0b1e80 2024-07-26T06:14:57.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.978+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb83c146500 con 0x7fb83c0b1e80 2024-07-26T06:14:57.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.980+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fb82401a070 con 0x7fb83c0b1e80 2024-07-26T06:14:57.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.981+0000 7fb833e00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb818076480 0x7fb818078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:57.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.981+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7fb824096b80 con 0x7fb83c0b1e80 2024-07-26T06:14:57.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.981+0000 7fb83b400640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb818076480 0x7fb818078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:57.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.982+0000 7fb83b400640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb818076480 0x7fb818078940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb83c15b8f0 tx=0x7fb82c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:57.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:57.985+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb83c146500 con 0x7fb83c0b1e80 2024-07-26T06:14:58.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.264+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fb83c0b57b0 con 0x7fb83c0b1e80 2024-07-26T06:14:58.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.307+0000 7fb833e00640 1 -- 172.21.15.163:0/2594491006 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v95) ==== 158+0+0 (secure 0 0 0) 0x7fb824060560 con 0x7fb83c0b1e80 2024-07-26T06:14:58.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb818076480 msgr2=0x7fb818078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb818076480 0x7fb818078940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb83c15b8f0 tx=0x7fb82c009040 comp rx=0 tx=0).stop 2024-07-26T06:14:58.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 msgr2=0x7fb83c1f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 0x7fb83c1f7ab0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb82400c3a0 tx=0x7fb82400c860 comp rx=0 tx=0).stop 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 shutdown_connections 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb818076480 0x7fb818078940 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c15d5a0 0x7fb83c145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0b23d0 0x7fb83c15df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 --2- 172.21.15.163:0/2594491006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb83c0b1e80 0x7fb83c1f7ab0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 >> 172.21.15.163:0/2594491006 conn(0x7fb83c0cfda0 msgr2=0x7fb83c0ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 shutdown_connections 2024-07-26T06:14:58.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.310+0000 7fb841c00640 1 -- 172.21.15.163:0/2594491006 wait complete. 2024-07-26T06:14:58.332 INFO:tasks.cephfs_test_runner:Validate binary .meta file on legacy subvol root doesn't break the system ... ok 2024-07-26T06:14:58.333 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:14:58.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.535+0000 7f8882600640 1 -- 172.21.15.163:0/921464634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8864004d40 msgr2=0x7f88640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.535+0000 7f8882600640 1 --2- 172.21.15.163:0/921464634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8864004d40 0x7f88640051c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8874009ed0 tx=0x7f8864005830 comp rx=0 tx=0).stop 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.536+0000 7f8881c00640 1 -- 172.21.15.163:0/921464634 reap_dead start 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.536+0000 7f8882600640 1 -- 172.21.15.163:0/921464634 shutdown_connections 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.536+0000 7f8882600640 1 -- 172.21.15.163:0/921464634 >> 172.21.15.163:0/921464634 conn(0x7f887c0b4ef0 msgr2=0x7f887c0b5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.536+0000 7f8882600640 1 -- 172.21.15.163:0/921464634 shutdown_connections 2024-07-26T06:14:58.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.536+0000 7f8882600640 1 -- 172.21.15.163:0/921464634 wait complete. 2024-07-26T06:14:58.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.537+0000 7f8882600640 1 Processor -- start 2024-07-26T06:14:58.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.537+0000 7f8882600640 1 -- start start 2024-07-26T06:14:58.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.538+0000 7f8882600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8864004d40 0x7f887c1f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.538+0000 7f8882600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f887c15f970 0x7f887c15e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.538+0000 7f8882600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 0x7f887c13e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.538+0000 7f8882600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f887c06da10 con 0x7f887c13fbe0 2024-07-26T06:14:58.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.538+0000 7f8882600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f887c06d890 con 0x7f8864004d40 2024-07-26T06:14:58.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.538+0000 7f8882600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f887c15d860 con 0x7f887c15f970 2024-07-26T06:14:58.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 0x7f887c13e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 0x7f887c13e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36956/0 (socket says 172.21.15.163:36956) 2024-07-26T06:14:58.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 -- 172.21.15.163:0/806726105 learned_addr learned my addr 172.21.15.163:0/806726105 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:58.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f887be00640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f887c15f970 0x7f887c15e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881200640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8864004d40 0x7f887c1f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 -- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f887c15f970 msgr2=0x7f887c15e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f887c15f970 0x7f887c15e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 -- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8864004d40 msgr2=0x7f887c1f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8864004d40 0x7f887c1f48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 -- 172.21.15.163:0/806726105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c15fdf0 con 0x7f887c13fbe0 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.539+0000 7f8881c00640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 0x7f887c13e080 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8874036040 tx=0x7f8874033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.540+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874043070 con 0x7f887c13fbe0 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.540+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8874033d20 con 0x7f887c13fbe0 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.540+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887c1600c0 con 0x7f887c13fbe0 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.540+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874038040 con 0x7f887c13fbe0 2024-07-26T06:14:58.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.540+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887c15dc90 con 0x7f887c13fbe0 2024-07-26T06:14:58.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.541+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887c0d26b0 con 0x7f887c13fbe0 2024-07-26T06:14:58.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.542+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f8874003bd0 con 0x7f887c13fbe0 2024-07-26T06:14:58.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.543+0000 7f8878c00640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8860076480 0x7f8860078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.543+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f88740b7790 con 0x7f887c13fbe0 2024-07-26T06:14:58.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.543+0000 7f8881200640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8860076480 0x7f8860078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.544+0000 7f8881200640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8860076480 0x7f8860078940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8870001860 tx=0x7f8870001c40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:58.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.547+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88740809f0 con 0x7f887c13fbe0 2024-07-26T06:14:58.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.753+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f887c06e780 con 0x7f887c13fbe0 2024-07-26T06:14:58.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.755+0000 7f8878c00640 1 -- 172.21.15.163:0/806726105 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+13044 (secure 0 0 0) 0x7f8874084cf0 con 0x7f887c13fbe0 2024-07-26T06:14:58.756 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:14:58.756 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":37,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:14:57.288767+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:6835/941040757":"2024-07-27T06:14:54.270441+0000","172.21.15.163:6833/941040757":"2024-07-27T06:14:54.270441+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8860076480 msgr2=0x7f8860078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8860076480 0x7f8860078940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8870001860 tx=0x7f8870001c40 comp rx=0 tx=0).stop 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 msgr2=0x7f887c13e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 0x7f887c13e080 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8874036040 tx=0x7f8874033c30 comp rx=0 tx=0).stop 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 shutdown_connections 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8860076480 0x7f8860078940 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f887c13fbe0 0x7f887c13e080 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f887c15f970 0x7f887c15e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 --2- 172.21.15.163:0/806726105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8864004d40 0x7f887c1f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 >> 172.21.15.163:0/806726105 conn(0x7f887c0b4ef0 msgr2=0x7f887c145a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:58.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 shutdown_connections 2024-07-26T06:14:58.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.758+0000 7f8882600640 1 -- 172.21.15.163:0/806726105 wait complete. 2024-07-26T06:14:58.778 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:13:21.677101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:14:58.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T06:14:58.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 -- 172.21.15.163:0/1421956960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d03d0 msgr2=0x7f886c0b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 --2- 172.21.15.163:0/1421956960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d03d0 0x7f886c0b6500 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8864005fd0 tx=0x7f886402aa80 comp rx=0 tx=0).stop 2024-07-26T06:14:58.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 -- 172.21.15.163:0/1421956960 shutdown_connections 2024-07-26T06:14:58.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 --2- 172.21.15.163:0/1421956960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d03d0 0x7f886c0b6500 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 --2- 172.21.15.163:0/1421956960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f886c15d3f0 0x7f886c0d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 -- 172.21.15.163:0/1421956960 >> 172.21.15.163:0/1421956960 conn(0x7f886c06d880 msgr2=0x7f886c06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:58.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 -- 172.21.15.163:0/1421956960 shutdown_connections 2024-07-26T06:14:58.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.985+0000 7f886be00640 1 -- 172.21.15.163:0/1421956960 wait complete. 2024-07-26T06:14:58.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.987+0000 7f886be00640 1 Processor -- start 2024-07-26T06:14:58.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.987+0000 7f886be00640 1 -- start start 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f886c0d03d0 0x7f886c0b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 0x7f886c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f886c15d3f0 0x7f886c0ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f886c0b6ee0 con 0x7f886c0d2ca0 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f886c0b6d60 con 0x7f886c15d3f0 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f886c0b7060 con 0x7f886c0d03d0 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f8863e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 0x7f886c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f8863e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 0x7f886c0b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36978/0 (socket says 172.21.15.163:36978) 2024-07-26T06:14:58.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f8863e00640 1 -- 172.21.15.163:0/2330880450 learned_addr learned my addr 172.21.15.163:0/2330880450 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.988+0000 7f886aa00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f886c0d03d0 0x7f886c0b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f886b400640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f886c15d3f0 0x7f886c0ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863e00640 1 -- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f886c0d03d0 msgr2=0x7f886c0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863e00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f886c0d03d0 0x7f886c0b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863e00640 1 -- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f886c15d3f0 msgr2=0x7f886c0ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863e00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f886c15d3f0 0x7f886c0ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863e00640 1 -- 172.21.15.163:0/2330880450 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c0b52a0 con 0x7f886c0d2ca0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863e00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 0x7f886c0b42f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f885400c460 tx=0x7f885400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854016020 con 0x7f886c0d2ca0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8854012960 con 0x7f886c0d2ca0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c0b3810 con 0x7f886c0d2ca0 2024-07-26T06:14:58.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854012db0 con 0x7f886c0d2ca0 2024-07-26T06:14:58.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.989+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c156e60 con 0x7f886c0d2ca0 2024-07-26T06:14:58.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.991+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886c0b7510 con 0x7f886c0d2ca0 2024-07-26T06:14:58.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.992+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f8854019020 con 0x7f886c0d2ca0 2024-07-26T06:14:58.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.993+0000 7f8863400640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8848076480 0x7f8848078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:58.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.993+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f88540978e0 con 0x7f886c0d2ca0 2024-07-26T06:14:58.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.993+0000 7f886aa00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8848076480 0x7f8848078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:58.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.994+0000 7f886aa00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8848076480 0x7f8848078940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f885c005d70 tx=0x7f885c005d00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:58.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:58.998+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8854020020 con 0x7f886c0d2ca0 2024-07-26T06:14:59.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.188+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f886c069930 con 0x7f886c0d2ca0 2024-07-26T06:14:59.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.188+0000 7f8863400640 1 -- 172.21.15.163:0/2330880450 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f8854061230 con 0x7f886c0d2ca0 2024-07-26T06:14:59.189 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-07-26T06:14:59.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8848076480 msgr2=0x7f8848078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8848076480 0x7f8848078940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f885c005d70 tx=0x7f885c005d00 comp rx=0 tx=0).stop 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 msgr2=0x7f886c0b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 0x7f886c0b42f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f885400c460 tx=0x7f885400c920 comp rx=0 tx=0).stop 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 shutdown_connections 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8848076480 0x7f8848078940 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f886c15d3f0 0x7f886c0ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f886c0d2ca0 0x7f886c0b42f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 --2- 172.21.15.163:0/2330880450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f886c0d03d0 0x7f886c0b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.191+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 >> 172.21.15.163:0/2330880450 conn(0x7f886c06d880 msgr2=0x7f886c0c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.192+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 shutdown_connections 2024-07-26T06:14:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.192+0000 7f886be00640 1 -- 172.21.15.163:0/2330880450 wait complete. 2024-07-26T06:14:59.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-07-26T06:14:59.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 -- 172.21.15.163:0/1474888096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76e0003a00 msgr2=0x7f76e0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:59.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 --2- 172.21.15.163:0/1474888096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76e0003a00 0x7f76e0003e00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00c2e0 tx=0x7f76e0005830 comp rx=0 tx=0).stop 2024-07-26T06:14:59.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 -- 172.21.15.163:0/1474888096 shutdown_connections 2024-07-26T06:14:59.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 --2- 172.21.15.163:0/1474888096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76e0003a00 0x7f76e0003e00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 -- 172.21.15.163:0/1474888096 >> 172.21.15.163:0/1474888096 conn(0x7f76fc0b0e30 msgr2=0x7f76fc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:14:59.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 -- 172.21.15.163:0/1474888096 shutdown_connections 2024-07-26T06:14:59.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.408+0000 7f7700e00640 1 -- 172.21.15.163:0/1474888096 wait complete. 2024-07-26T06:14:59.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.410+0000 7f7700e00640 1 Processor -- start 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.410+0000 7f7700e00640 1 -- start start 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f7700e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76e0003a00 0x7f76fc072240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f7700e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 0x7f76fc14b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f7700e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76fc154ce0 0x7f76fc14bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f7700e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc159750 con 0x7f76fc154860 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f7700e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc06f500 con 0x7f76e0003a00 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f7700e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f76fc1598d0 con 0x7f76fc154ce0 2024-07-26T06:14:59.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f76f3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 0x7f76fc14b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f76f3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 0x7f76fc14b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37000/0 (socket says 172.21.15.163:37000) 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.411+0000 7f76f3e00640 1 -- 172.21.15.163:0/861623247 learned_addr learned my addr 172.21.15.163:0/861623247 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3e00640 1 -- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76fc154ce0 msgr2=0x7f76fc14bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76fc154ce0 0x7f76fc14bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3e00640 1 -- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76e0003a00 msgr2=0x7f76fc072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76e0003a00 0x7f76fc072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3e00640 1 -- 172.21.15.163:0/861623247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc1513a0 con 0x7f76fc154860 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 0x7f76fc14b620 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f76e400c3f0 tx=0x7f76e400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4011790 con 0x7f76fc154860 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76e4017070 con 0x7f76fc154860 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76fc151680 con 0x7f76fc154860 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.412+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4011cc0 con 0x7f76fc154860 2024-07-26T06:14:59.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.413+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76fc151c40 con 0x7f76fc154860 2024-07-26T06:14:59.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.414+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc0cd530 con 0x7f76fc154860 2024-07-26T06:14:59.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.415+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f76e401c460 con 0x7f76fc154860 2024-07-26T06:14:59.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.416+0000 7f76f3400640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f76d80764b0 0x7f76d8078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:14:59.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.416+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f76e4097650 con 0x7f76fc154860 2024-07-26T06:14:59.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.416+0000 7f76fb400640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f76d80764b0 0x7f76d8078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:14:59.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.417+0000 7f76fb400640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f76d80764b0 0x7f76d8078970 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f76ec033000 tx=0x7f76ec001a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:14:59.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.420+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76e4061050 con 0x7f76fc154860 2024-07-26T06:14:59.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:14:59.701+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7f76fc15d250 con 0x7f76fc154860 2024-07-26T06:15:00.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.313+0000 7f76f3400640 1 -- 172.21.15.163:0/861623247 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v97) ==== 143+0+0 (secure 0 0 0) 0x7f76e4069040 con 0x7f76fc154860 2024-07-26T06:15:00.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f76d80764b0 msgr2=0x7f76d8078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f76d80764b0 0x7f76d8078970 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f76ec033000 tx=0x7f76ec001a30 comp rx=0 tx=0).stop 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 msgr2=0x7f76fc14b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 0x7f76fc14b620 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f76e400c3f0 tx=0x7f76e400c7f0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 shutdown_connections 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f76d80764b0 0x7f76d8078970 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76fc154ce0 0x7f76fc14bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76fc154860 0x7f76fc14b620 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 --2- 172.21.15.163:0/861623247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76e0003a00 0x7f76fc072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 >> 172.21.15.163:0/861623247 conn(0x7f76fc0b0e30 msgr2=0x7f76fc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:00.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.318+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 shutdown_connections 2024-07-26T06:15:00.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.319+0000 7f7700e00640 1 -- 172.21.15.163:0/861623247 wait complete. 2024-07-26T06:15:00.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-07-26T06:15:00.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 -- 172.21.15.163:0/3906714628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f90159370 msgr2=0x7f3f90159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 --2- 172.21.15.163:0/3906714628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f90159370 0x7f3f90159750 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009830 tx=0x7f3f8402a510 comp rx=0 tx=0).stop 2024-07-26T06:15:00.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 -- 172.21.15.163:0/3906714628 shutdown_connections 2024-07-26T06:15:00.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 --2- 172.21.15.163:0/3906714628 >> v2:172.21.15.164:3300/0 conn(0x7f3f900b4420 0x7f3f900b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 --2- 172.21.15.163:0/3906714628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f90159370 0x7f3f90159750 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 -- 172.21.15.163:0/3906714628 >> 172.21.15.163:0/3906714628 conn(0x7f3f900b0e30 msgr2=0x7f3f900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:00.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 -- 172.21.15.163:0/3906714628 shutdown_connections 2024-07-26T06:15:00.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.547+0000 7f3f96c00640 1 -- 172.21.15.163:0/3906714628 wait complete. 2024-07-26T06:15:00.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.549+0000 7f3f96c00640 1 Processor -- start 2024-07-26T06:15:00.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.549+0000 7f3f96c00640 1 -- start start 2024-07-26T06:15:00.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f96c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 0x7f3f9014bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:00.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f96c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f90159370 0x7f3f9015c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:00.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f96c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015a990 0x7f3f900ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:00.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f96c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9006f2f0 con 0x7f3f900b4420 2024-07-26T06:15:00.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f96c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f900cd210 con 0x7f3f90159370 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f96c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f9006f470 con 0x7f3f9015a990 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f95800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 0x7f3f9014bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f95800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 0x7f3f9014bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37006/0 (socket says 172.21.15.163:37006) 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.550+0000 7f3f95800640 1 -- 172.21.15.163:0/1889591360 learned_addr learned my addr 172.21.15.163:0/1889591360 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f94e00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f90159370 0x7f3f9015c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f96200640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015a990 0x7f3f900ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f95800640 1 -- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015a990 msgr2=0x7f3f900ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f95800640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015a990 0x7f3f900ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f95800640 1 -- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f90159370 msgr2=0x7f3f9015c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f95800640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f90159370 0x7f3f9015c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f95800640 1 -- 172.21.15.163:0/1889591360 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9015af00 con 0x7f3f900b4420 2024-07-26T06:15:00.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f95800640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 0x7f3f9014bce0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000c00 tx=0x7f3f84033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84031d60 con 0x7f3f900b4420 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f84033e10 con 0x7f3f900b4420 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840332b0 con 0x7f3f900b4420 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.551+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9015b1e0 con 0x7f3f900b4420 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.552+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9015b7a0 con 0x7f3f900b4420 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.553+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f900b2130 con 0x7f3f900b4420 2024-07-26T06:15:00.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.554+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f3f84033450 con 0x7f3f900b4420 2024-07-26T06:15:00.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.555+0000 7f3f8d600640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3f70076480 0x7f3f70078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:00.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.555+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f3f840b8b00 con 0x7f3f900b4420 2024-07-26T06:15:00.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.555+0000 7f3f94e00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3f70076480 0x7f3f70078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:00.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.557+0000 7f3f94e00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3f70076480 0x7f3f70078940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3f901f7970 tx=0x7f3f7c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:00.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.559+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3f8403f030 con 0x7f3f900b4420 2024-07-26T06:15:00.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.768+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f3f90159a70 con 0x7f3f900b4420 2024-07-26T06:15:00.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.769+0000 7f3f8d600640 1 -- 172.21.15.163:0/1889591360 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v29) ==== 59+0+1998 (secure 0 0 0) 0x7f3f840b8e20 con 0x7f3f900b4420 2024-07-26T06:15:00.770 INFO:teuthology.orchestra.run.smithi163.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-07-26T06:15:00.770 INFO:teuthology.orchestra.run.smithi163.stdout:mon advanced mon_op_complaint_time 120 2024-07-26T06:15:00.770 INFO:teuthology.orchestra.run.smithi163.stdout:mon dev mon_warn_on_pool_no_app false 2024-07-26T06:15:00.770 INFO:teuthology.orchestra.run.smithi163.stdout:osd advanced osd_op_complaint_time 180.000000 2024-07-26T06:15:00.770 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 5439.226762 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3901.149445 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5414.054049 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4219.540367 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4287.026771 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4403.390021 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3932.844298 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4626.659828 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds 20/20 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds_balancer 20/20 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_ms 1/1 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_frag true 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_scatterstat true 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced mds_op_complaint_time 180.000000 2024-07-26T06:15:00.771 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_verify_scatter true 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced osd_op_complaint_time 180.000000 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_mon_op_timeout 900 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_osd_op_timeout 900 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced client_mount_timeout 600 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_client 20/20 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_ms 1/1 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_mon_op_timeout 900 2024-07-26T06:15:00.772 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_osd_op_timeout 900 2024-07-26T06:15:00.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3f70076480 msgr2=0x7f3f70078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3f70076480 0x7f3f70078940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3f901f7970 tx=0x7f3f7c009040 comp rx=0 tx=0).stop 2024-07-26T06:15:00.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 msgr2=0x7f3f9014bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:00.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 0x7f3f9014bce0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000c00 tx=0x7f3f84033c20 comp rx=0 tx=0).stop 2024-07-26T06:15:00.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 shutdown_connections 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3f70076480 0x7f3f70078940 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015a990 0x7f3f900ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f90159370 0x7f3f9015c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 --2- 172.21.15.163:0/1889591360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f900b4420 0x7f3f9014bce0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 >> 172.21.15.163:0/1889591360 conn(0x7f3f900b0e30 msgr2=0x7f3f90073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 shutdown_connections 2024-07-26T06:15:00.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:00.772+0000 7f3f96c00640 1 -- 172.21.15.163:0/1889591360 wait complete. 2024-07-26T06:15:00.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.004+0000 7f792c600640 1 -- 172.21.15.163:0/4036850715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79240b2f90 msgr2=0x7f7924159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 --2- 172.21.15.163:0/4036850715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79240b2f90 0x7f7924159380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79240b47b0 tx=0x7f791002a4d0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 -- 172.21.15.163:0/4036850715 shutdown_connections 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 --2- 172.21.15.163:0/4036850715 >> v2:172.21.15.164:3300/0 conn(0x7f79240cc8e0 0x7f79240ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 --2- 172.21.15.163:0/4036850715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79240b2f90 0x7f7924159380 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 -- 172.21.15.163:0/4036850715 >> 172.21.15.163:0/4036850715 conn(0x7f792406d420 msgr2=0x7f792406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 -- 172.21.15.163:0/4036850715 shutdown_connections 2024-07-26T06:15:01.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.005+0000 7f792c600640 1 -- 172.21.15.163:0/4036850715 wait complete. 2024-07-26T06:15:01.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.006+0000 7f792c600640 1 Processor -- start 2024-07-26T06:15:01.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.007+0000 7f792c600640 1 -- start start 2024-07-26T06:15:01.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.007+0000 7f792c600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 0x7f79240af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.007+0000 7f792c600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240b2f90 0x7f79240afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f792c600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79240cc8e0 0x7f7924149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f792c600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79240cd8f0 con 0x7f792406f350 2024-07-26T06:15:01.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f792c600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f79240cd770 con 0x7f79240b2f90 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f792c600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f79240cda70 con 0x7f79240cc8e0 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 0x7f79240af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 0x7f79240af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37020/0 (socket says 172.21.15.163:37020) 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 -- 172.21.15.163:0/3971055281 learned_addr learned my addr 172.21.15.163:0/3971055281 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f792a600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79240cc8e0 0x7f7924149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929200640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240b2f90 0x7f79240afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 -- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79240cc8e0 msgr2=0x7f7924149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79240cc8e0 0x7f7924149ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 -- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240b2f90 msgr2=0x7f79240afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240b2f90 0x7f79240afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.008+0000 7f7929c00640 1 -- 172.21.15.163:0/3971055281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79240b0420 con 0x7f792406f350 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.009+0000 7f7929c00640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 0x7f79240af810 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f791800c410 tx=0x7f791800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.009+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918017070 con 0x7f792406f350 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.009+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79180035f0 con 0x7f792406f350 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.009+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918003a80 con 0x7f792406f350 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.009+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79240b0700 con 0x7f792406f350 2024-07-26T06:15:01.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.009+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7924139440 con 0x7f792406f350 2024-07-26T06:15:01.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.010+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7924159380 con 0x7f792406f350 2024-07-26T06:15:01.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.011+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f7918003c20 con 0x7f792406f350 2024-07-26T06:15:01.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.012+0000 7f7921600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7904076480 0x7f7904078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.012+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f7918096b70 con 0x7f792406f350 2024-07-26T06:15:01.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.013+0000 7f7929200640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7904076480 0x7f7904078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.014+0000 7f7929200640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7904076480 0x7f7904078940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7910033000 tx=0x7f7910004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:01.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.016+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7918060420 con 0x7f792406f350 2024-07-26T06:15:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.220+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f79240b0c20 con 0x7f792406f350 2024-07-26T06:15:01.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.225+0000 7f7921600640 1 -- 172.21.15.163:0/3971055281 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f791805fd20 con 0x7f792406f350 2024-07-26T06:15:01.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7904076480 msgr2=0x7f7904078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7904076480 0x7f7904078940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7910033000 tx=0x7f7910004730 comp rx=0 tx=0).stop 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 msgr2=0x7f79240af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 0x7f79240af810 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f791800c410 tx=0x7f791800c8d0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 shutdown_connections 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7904076480 0x7f7904078940 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79240cc8e0 0x7f7924149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240b2f90 0x7f79240afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 --2- 172.21.15.163:0/3971055281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f792406f350 0x7f79240af810 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 >> 172.21.15.163:0/3971055281 conn(0x7f792406d420 msgr2=0x7f792414dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 shutdown_connections 2024-07-26T06:15:01.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.228+0000 7f792c600640 1 -- 172.21.15.163:0/3971055281 wait complete. 2024-07-26T06:15:01.248 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:15:01.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:15:01.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:15:01.497 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:15:01.498 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:15:01.498 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:01.555 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:15:01.555 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:15:01.555 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:01.613 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:15:01.613 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:15:01.614 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:01.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:15:01.671 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:15:01.671 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:01.729 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:15:01.729 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-07-26T06:15:01.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.938+0000 7f1516200640 1 -- 172.21.15.163:0/2161995137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15101556f0 msgr2=0x7f15100cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.938+0000 7f1516200640 1 --2- 172.21.15.163:0/2161995137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15101556f0 0x7f15100cdd90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1504009830 tx=0x7f150402a500 comp rx=0 tx=0).stop 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.938+0000 7f1516200640 1 -- 172.21.15.163:0/2161995137 shutdown_connections 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.938+0000 7f1516200640 1 --2- 172.21.15.163:0/2161995137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1510071c40 0x7f15100b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.938+0000 7f1516200640 1 --2- 172.21.15.163:0/2161995137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15101556f0 0x7f15100cdd90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.938+0000 7f1516200640 1 -- 172.21.15.163:0/2161995137 >> 172.21.15.163:0/2161995137 conn(0x7f15100b0e30 msgr2=0x7f15100b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.939+0000 7f1516200640 1 -- 172.21.15.163:0/2161995137 shutdown_connections 2024-07-26T06:15:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.939+0000 7f1516200640 1 -- 172.21.15.163:0/2161995137 wait complete. 2024-07-26T06:15:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.940+0000 7f1516200640 1 Processor -- start 2024-07-26T06:15:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.940+0000 7f1516200640 1 -- start start 2024-07-26T06:15:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f1516200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1510071c40 0x7f1510149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f1516200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 0x7f15101547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f1516200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15101556f0 0x7f15101f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f1516200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15100b3fe0 con 0x7f15100ce2d0 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f1516200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f15100b3e60 con 0x7f1510071c40 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f1516200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f15100b4160 con 0x7f15101556f0 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f150fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 0x7f15101547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f150fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 0x7f15101547e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37024/0 (socket says 172.21.15.163:37024) 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.941+0000 7f150fe00640 1 -- 172.21.15.163:0/799937331 learned_addr learned my addr 172.21.15.163:0/799937331 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150fe00640 1 -- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15101556f0 msgr2=0x7f15101f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150fe00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15101556f0 0x7f15101f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150fe00640 1 -- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1510071c40 msgr2=0x7f1510149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f1514e00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1510071c40 0x7f1510149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150fe00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1510071c40 0x7f1510149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150fe00640 1 -- 172.21.15.163:0/799937331 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1510149e00 con 0x7f15100ce2d0 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150fe00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 0x7f15101547e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00c500 tx=0x7f14fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc017070 con 0x7f15100ce2d0 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14fc0035f0 con 0x7f15100ce2d0 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc003a40 con 0x7f15100ce2d0 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.942+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151014a0e0 con 0x7f15100ce2d0 2024-07-26T06:15:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.943+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151014a6a0 con 0x7f15100ce2d0 2024-07-26T06:15:01.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.944+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151015e680 con 0x7f15100ce2d0 2024-07-26T06:15:01.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.945+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f14fc01a070 con 0x7f15100ce2d0 2024-07-26T06:15:01.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.946+0000 7f150cc00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14f0076480 0x7f14f0078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:01.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.946+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f14fc096cf0 con 0x7f15100ce2d0 2024-07-26T06:15:01.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.946+0000 7f1514e00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14f0076480 0x7f14f0078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:01.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.947+0000 7f1514e00640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14f0076480 0x7f14f0078940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1504034000 tx=0x7f1504004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:01.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:01.950+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f151015e680 con 0x7f15100ce2d0 2024-07-26T06:15:02.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.186+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1510154d20 con 0x7f15100ce2d0 2024-07-26T06:15:02.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.187+0000 7f150cc00640 1 -- 172.21.15.163:0/799937331 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3920 (secure 0 0 0) 0x7f14fc0606c0 con 0x7f15100ce2d0 2024-07-26T06:15:02.188 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:02.188 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":19,"btime":"2024-07-26T06:14:55:274136+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4562,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2367823484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2367823484},{"type":"v1","addr":"172.21.15.164:6836","nonce":2367823484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2795042464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2795042464},{"type":"v1","addr":"172.21.15.164:6837","nonce":2795042464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/254760042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":254760042},{"type":"v1","addr":"172.21.15.163:6834","nonce":254760042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5154,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3689105075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3689105075},{"type":"v1","addr":"172.21.15.163:6835","nonce":3689105075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-07-26T06:15:02.188 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 19 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14f0076480 msgr2=0x7f14f0078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14f0076480 0x7f14f0078940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1504034000 tx=0x7f1504004730 comp rx=0 tx=0).stop 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 msgr2=0x7f15101547e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 0x7f15101547e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00c500 tx=0x7f14fc00c9c0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 shutdown_connections 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f14f0076480 0x7f14f0078940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15101556f0 0x7f15101f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15100ce2d0 0x7f15101547e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 --2- 172.21.15.163:0/799937331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1510071c40 0x7f1510149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 >> 172.21.15.163:0/799937331 conn(0x7f15100b0e30 msgr2=0x7f151015b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:02.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 shutdown_connections 2024-07-26T06:15:02.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.190+0000 7f1516200640 1 -- 172.21.15.163:0/799937331 wait complete. 2024-07-26T06:15:02.212 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:15:02.212 INFO:tasks.ceph.mds.a:Stopping old one... 2024-07-26T06:15:02.212 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:15:02.212 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:15:02.213 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:15:02.213 INFO:tasks.ceph.mds.c:Stopping old one... 2024-07-26T06:15:02.213 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:15:02.213 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:15:02.213 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:15:02.213 INFO:tasks.ceph.mds.b:Stopping old one... 2024-07-26T06:15:02.213 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:15:02.214 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:15:02.214 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:15:02.214 INFO:tasks.ceph.mds.d:Stopping old one... 2024-07-26T06:15:02.214 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:15:02.214 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:15:02.235 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:15:02.235 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:15:02.237 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:15:02.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:15:02.239 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:15:02.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:15:02.242 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:15:02.242 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:15:02.285 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:15:02.286 DEBUG:teuthology.parallel:result is None 2024-07-26T06:15:02.289 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:15:02.290 DEBUG:teuthology.parallel:result is None 2024-07-26T06:15:02.292 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:15:02.292 DEBUG:teuthology.parallel:result is None 2024-07-26T06:15:02.296 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:15:02.296 DEBUG:teuthology.parallel:result is None 2024-07-26T06:15:02.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-07-26T06:15:02.436 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:15:02.437 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:15:02.456 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:15:02.462 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.477+0000 7f15eb000640 1 -- 172.21.15.163:0/2851487345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e40cb7e0 msgr2=0x7f15e415ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.477+0000 7f15eb000640 1 --2- 172.21.15.163:0/2851487345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e40cb7e0 0x7f15e415ed20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009830 tx=0x7f15d402a530 comp rx=0 tx=0).stop 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.478+0000 7f15eb000640 1 -- 172.21.15.163:0/2851487345 shutdown_connections 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.478+0000 7f15eb000640 1 --2- 172.21.15.163:0/2851487345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e40cb7e0 0x7f15e415ed20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.478+0000 7f15eb000640 1 -- 172.21.15.163:0/2851487345 >> 172.21.15.163:0/2851487345 conn(0x7f15e406d420 msgr2=0x7f15e406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.478+0000 7f15eb000640 1 -- 172.21.15.163:0/2851487345 shutdown_connections 2024-07-26T06:15:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.478+0000 7f15eb000640 1 -- 172.21.15.163:0/2851487345 wait complete. 2024-07-26T06:15:02.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.479+0000 7f15eb000640 1 Processor -- start 2024-07-26T06:15:02.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.479+0000 7f15eb000640 1 -- start start 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.480+0000 7f15eb000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 0x7f15e4150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.480+0000 7f15eb000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e4071cf0 0x7f15e4159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.480+0000 7f15eb000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e40cb7e0 0x7f15e41f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.480+0000 7f15eb000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15e40738c0 con 0x7f15e406bf40 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.480+0000 7f15eb000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4073740 con 0x7f15e4071cf0 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.480+0000 7f15eb000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f15e4073a40 con 0x7f15e40cb7e0 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 0x7f15e4150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 0x7f15e4150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37056/0 (socket says 172.21.15.163:37056) 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 -- 172.21.15.163:0/3741187797 learned_addr learned my addr 172.21.15.163:0/3741187797 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e9000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e40cb7e0 0x7f15e41f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3400640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e4071cf0 0x7f15e4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 -- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e40cb7e0 msgr2=0x7f15e41f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e40cb7e0 0x7f15e41f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 -- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e4071cf0 msgr2=0x7f15e4159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e4071cf0 0x7f15e4159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 -- 172.21.15.163:0/3741187797 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e41512f0 con 0x7f15e406bf40 2024-07-26T06:15:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.481+0000 7f15e3e00640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 0x7f15e4150db0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f15d402ab20 tx=0x7f15d4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.482+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4031d60 con 0x7f15e406bf40 2024-07-26T06:15:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.482+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15d4033e10 con 0x7f15e406bf40 2024-07-26T06:15:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.482+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e415a0e0 con 0x7f15e406bf40 2024-07-26T06:15:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.482+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4005260 con 0x7f15e406bf40 2024-07-26T06:15:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.482+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e415a6a0 con 0x7f15e406bf40 2024-07-26T06:15:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.483+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e415ed20 con 0x7f15e406bf40 2024-07-26T06:15:02.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.484+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f15d4005400 con 0x7f15e406bf40 2024-07-26T06:15:02.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.486+0000 7f15dbe00640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f15c4076480 0x7f15c4078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:02.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.486+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4943+0+0 (secure 0 0 0) 0x7f15d40b8860 con 0x7f15e406bf40 2024-07-26T06:15:02.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.486+0000 7f15e3400640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f15c4076480 0x7f15c4078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:02.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.487+0000 7f15e3400640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f15c4076480 0x7f15c4078940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f15e406ca20 tx=0x7f15d0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:02.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.490+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f15d403f030 con 0x7f15e406bf40 2024-07-26T06:15:02.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:02.708+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f15e4069050 con 0x7f15e406bf40 2024-07-26T06:15:03.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.346+0000 7f15dbe00640 1 -- 172.21.15.163:0/3741187797 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v39) ==== 97+0+0 (secure 0 0 0) 0x7f15d4082090 con 0x7f15e406bf40 2024-07-26T06:15:03.347 INFO:teuthology.orchestra.run.smithi163.stderr: removed all blocklist entries 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.352+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f15c4076480 msgr2=0x7f15c4078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.352+0000 7f15eb000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f15c4076480 0x7f15c4078940 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f15e406ca20 tx=0x7f15d0009040 comp rx=0 tx=0).stop 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.352+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 msgr2=0x7f15e4150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.352+0000 7f15eb000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 0x7f15e4150db0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f15d402ab20 tx=0x7f15d4033c20 comp rx=0 tx=0).stop 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 shutdown_connections 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f15c4076480 0x7f15c4078940 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e40cb7e0 0x7f15e41f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e4071cf0 0x7f15e4159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 --2- 172.21.15.163:0/3741187797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15e406bf40 0x7f15e4150db0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 >> 172.21.15.163:0/3741187797 conn(0x7f15e406d420 msgr2=0x7f15e4073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:03.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 shutdown_connections 2024-07-26T06:15:03.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.353+0000 7f15eb000640 1 -- 172.21.15.163:0/3741187797 wait complete. 2024-07-26T06:15:03.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-07-26T06:15:03.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 -- 172.21.15.163:0/2763744263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 msgr2=0x7f29580b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 --2- 172.21.15.163:0/2763744263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580b6a00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2950005fd0 tx=0x7f295002a820 comp rx=0 tx=0).stop 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 -- 172.21.15.163:0/2763744263 shutdown_connections 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 --2- 172.21.15.163:0/2763744263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580b6a00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 --2- 172.21.15.163:0/2763744263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29580b1e80 0x7f29580b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 -- 172.21.15.163:0/2763744263 >> 172.21.15.163:0/2763744263 conn(0x7f29580cfda0 msgr2=0x7f29580d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 -- 172.21.15.163:0/2763744263 shutdown_connections 2024-07-26T06:15:03.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.577+0000 7f295ec00640 1 -- 172.21.15.163:0/2763744263 wait complete. 2024-07-26T06:15:03.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.579+0000 7f295ec00640 1 Processor -- start 2024-07-26T06:15:03.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.579+0000 7f295ec00640 1 -- start start 2024-07-26T06:15:03.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.580+0000 7f295ec00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29580b1e80 0x7f29581f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:03.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.580+0000 7f295ec00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.580+0000 7f295ec00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295815d5a0 0x7f29581579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.580+0000 7f295ec00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29580b4f20 con 0x7f29580b23d0 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.580+0000 7f295ec00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f29580b4da0 con 0x7f29580b1e80 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.580+0000 7f295ec00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f29580b50a0 con 0x7f295815d5a0 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37068/0 (socket says 172.21.15.163:37068) 2024-07-26T06:15:03.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 -- 172.21.15.163:0/1664638521 learned_addr learned my addr 172.21.15.163:0/1664638521 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:03.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 -- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295815d5a0 msgr2=0x7f29581579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295e200640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295815d5a0 0x7f29581579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295d800640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29580b1e80 0x7f29581f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295815d5a0 0x7f29581579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 -- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29580b1e80 msgr2=0x7f29581f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29580b1e80 0x7f29581f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295ce00640 1 -- 172.21.15.163:0/1664638521 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29580c9c80 con 0x7f29580b23d0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.581+0000 7f295d800640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29580b1e80 0x7f29581f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.582+0000 7f295ce00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580c9740 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f294400d7a0 tx=0x7f294400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.582+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944017070 con 0x7f29580b23d0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.582+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2944009d20 con 0x7f29580b23d0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.582+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29580c7e60 con 0x7f29580b23d0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.582+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944004280 con 0x7f29580b23d0 2024-07-26T06:15:03.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.582+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29580c8420 con 0x7f29580b23d0 2024-07-26T06:15:03.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.583+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29580b5590 con 0x7f29580b23d0 2024-07-26T06:15:03.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.584+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f29440035f0 con 0x7f29580b23d0 2024-07-26T06:15:03.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.585+0000 7f2955600640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2938076480 0x7f2938078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:03.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.585+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f2944097900 con 0x7f29580b23d0 2024-07-26T06:15:03.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.586+0000 7f295d800640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2938076480 0x7f2938078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:03.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.587+0000 7f295d800640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2938076480 0x7f2938078940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2948000f60 tx=0x7f2948009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:03.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.590+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2944061210 con 0x7f29580b23d0 2024-07-26T06:15:03.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.876+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f29580c6af0 con 0x7f29580b23d0 2024-07-26T06:15:03.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.877+0000 7f2955600640 1 -- 172.21.15.163:0/1664638521 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) ==== 80+0+5304 (secure 0 0 0) 0x7f2944069030 con 0x7f29580b23d0 2024-07-26T06:15:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_dump": [ 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.a", 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA==", 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.b", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg==", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.c", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw==", 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.d", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg==", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:15:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.0", 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA==", 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.1", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA==", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.2", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA==", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.3", 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg==", 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.4", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA==", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.5", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g==", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.6", 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ==", 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.7", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA==", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.0", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew==", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.1", 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ==", 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.2", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw==", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.3", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw==", 2024-07-26T06:15:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.admin", 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm+ibKBxAA1ZBGvVy5rNIzMWwOFwFJFw==", 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:15:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.x", 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ==", 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:15:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.y", 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ==", 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:15:03.892 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:15:03.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2938076480 msgr2=0x7f2938078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2938076480 0x7f2938078940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2948000f60 tx=0x7f2948009040 comp rx=0 tx=0).stop 2024-07-26T06:15:03.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 msgr2=0x7f29580c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:03.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580c9740 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f294400d7a0 tx=0x7f294400dc60 comp rx=0 tx=0).stop 2024-07-26T06:15:03.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 shutdown_connections 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2938076480 0x7f2938078940 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295815d5a0 0x7f29581579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29580b23d0 0x7f29580c9740 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 --2- 172.21.15.163:0/1664638521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29580b1e80 0x7f29581f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 >> 172.21.15.163:0/1664638521 conn(0x7f29580cfda0 msgr2=0x7f29580b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 shutdown_connections 2024-07-26T06:15:03.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:03.880+0000 7f295ec00640 1 -- 172.21.15.163:0/1664638521 wait complete. 2024-07-26T06:15:03.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:15:04.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 -- 172.21.15.163:0/1629155068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f741556f0 msgr2=0x7f4f740cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1629155068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f741556f0 0x7f4f740cdd90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4f64005fd0 tx=0x7f4f6402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 -- 172.21.15.163:0/1629155068 shutdown_connections 2024-07-26T06:15:04.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1629155068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f741556f0 0x7f4f740cdd90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 -- 172.21.15.163:0/1629155068 >> 172.21.15.163:0/1629155068 conn(0x7f4f740b0e30 msgr2=0x7f4f740b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:04.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 -- 172.21.15.163:0/1629155068 shutdown_connections 2024-07-26T06:15:04.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.097+0000 7f4f79a00640 1 -- 172.21.15.163:0/1629155068 wait complete. 2024-07-26T06:15:04.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.099+0000 7f4f79a00640 1 Processor -- start 2024-07-26T06:15:04.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.099+0000 7f4f79a00640 1 -- start start 2024-07-26T06:15:04.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 0x7f4f741e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f740ce2d0 0x7f4f7414f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f741556f0 0x7f4f741f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4f740b41f0 con 0x7f4f74071c40 2024-07-26T06:15:04.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4f740b4070 con 0x7f4f741556f0 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f740b4370 con 0x7f4f740ce2d0 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 0x7f4f741e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 0x7f4f741e5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37090/0 (socket says 172.21.15.163:37090) 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f73e00640 1 -- 172.21.15.163:0/1707239420 learned_addr learned my addr 172.21.15.163:0/1707239420 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f6be00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f740ce2d0 0x7f4f7414f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f73e00640 1 -- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f740ce2d0 msgr2=0x7f4f7414f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f79000640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f741556f0 0x7f4f741f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.100+0000 7f4f73e00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f740ce2d0 0x7f4f7414f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f73e00640 1 -- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f741556f0 msgr2=0x7f4f741f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f73e00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f741556f0 0x7f4f741f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f73e00640 1 -- 172.21.15.163:0/1707239420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f741e6780 con 0x7f4f74071c40 2024-07-26T06:15:04.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f79000640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f741556f0 0x7f4f741f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:04.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f73e00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 0x7f4f741e5f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006010 tx=0x7f4f640038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:04.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6403e030 con 0x7f4f74071c40 2024-07-26T06:15:04.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f64003ca0 con 0x7f4f74071c40 2024-07-26T06:15:04.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64035040 con 0x7f4f74071c40 2024-07-26T06:15:04.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f741e6a60 con 0x7f4f74071c40 2024-07-26T06:15:04.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.101+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f741e6470 con 0x7f4f74071c40 2024-07-26T06:15:04.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.102+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f740cdd90 con 0x7f4f74071c40 2024-07-26T06:15:04.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.103+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f4f64041030 con 0x7f4f74071c40 2024-07-26T06:15:04.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.104+0000 7f4f70c00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f50076480 0x7f4f50078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.104+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f4f640b7d60 con 0x7f4f74071c40 2024-07-26T06:15:04.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.105+0000 7f4f6be00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f50076480 0x7f4f50078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:04.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.106+0000 7f4f6be00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f50076480 0x7f4f50078940 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4f74159af0 tx=0x7f4f5c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:04.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.108+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f64081670 con 0x7f4f74071c40 2024-07-26T06:15:04.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.303+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f4f740ced60 con 0x7f4f74071c40 2024-07-26T06:15:04.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.304+0000 7f4f70c00640 1 -- 172.21.15.163:0/1707239420 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) ==== 84+0+65 (secure 0 0 0) 0x7f4f64080f70 con 0x7f4f74071c40 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:04.305 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:15:04.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f50076480 msgr2=0x7f4f50078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f50076480 0x7f4f50078940 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4f74159af0 tx=0x7f4f5c009040 comp rx=0 tx=0).stop 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 msgr2=0x7f4f741e5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 0x7f4f741e5f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006010 tx=0x7f4f640038a0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 shutdown_connections 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4f50076480 0x7f4f50078940 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f741556f0 0x7f4f741f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f740ce2d0 0x7f4f7414f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 --2- 172.21.15.163:0/1707239420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4f74071c40 0x7f4f741e5f30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 >> 172.21.15.163:0/1707239420 conn(0x7f4f740b0e30 msgr2=0x7f4f740b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.306+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 shutdown_connections 2024-07-26T06:15:04.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.307+0000 7f4f79a00640 1 -- 172.21.15.163:0/1707239420 wait complete. 2024-07-26T06:15:04.325 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:15:04.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:15:04.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.516+0000 7f96f6a00640 1 -- 172.21.15.163:0/3178143687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f015d5a0 msgr2=0x7f96f00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.516+0000 7f96f6a00640 1 --2- 172.21.15.163:0/3178143687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f015d5a0 0x7f96f00ce2f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009830 tx=0x7f96e402a480 comp rx=0 tx=0).stop 2024-07-26T06:15:04.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.517+0000 7f96f6a00640 1 -- 172.21.15.163:0/3178143687 shutdown_connections 2024-07-26T06:15:04.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.517+0000 7f96f6a00640 1 --2- 172.21.15.163:0/3178143687 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f015d5a0 0x7f96f00ce2f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.517+0000 7f96f6a00640 1 -- 172.21.15.163:0/3178143687 >> 172.21.15.163:0/3178143687 conn(0x7f96f00cfda0 msgr2=0x7f96f00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:04.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.517+0000 7f96f6a00640 1 -- 172.21.15.163:0/3178143687 shutdown_connections 2024-07-26T06:15:04.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.517+0000 7f96f6a00640 1 -- 172.21.15.163:0/3178143687 wait complete. 2024-07-26T06:15:04.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.518+0000 7f96f6a00640 1 Processor -- start 2024-07-26T06:15:04.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.518+0000 7f96f6a00640 1 -- start start 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.519+0000 7f96f6a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 0x7f96f00cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.519+0000 7f96f6a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f00ce830 0x7f96f01f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.519+0000 7f96f6a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f015d5a0 0x7f96f00c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.519+0000 7f96f6a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f96f00b6f60 con 0x7f96f00b1e80 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.519+0000 7f96f6a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96f00b6de0 con 0x7f96f015d5a0 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.519+0000 7f96f6a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96f00b54b0 con 0x7f96f00ce830 2024-07-26T06:15:04.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 0x7f96f00cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 0x7f96f00cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37108/0 (socket says 172.21.15.163:37108) 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 -- 172.21.15.163:0/662517133 learned_addr learned my addr 172.21.15.163:0/662517133 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 -- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f00ce830 msgr2=0x7f96f01f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f00ce830 0x7f96f01f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 -- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f015d5a0 msgr2=0x7f96f00c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f015d5a0 0x7f96f00c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 -- 172.21.15.163:0/662517133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f00ccb20 con 0x7f96f00b1e80 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.520+0000 7f96f5600640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 0x7f96f00cced0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009950 tx=0x7f96e4033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.521+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4031d60 con 0x7f96f00b1e80 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.521+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96e4033070 con 0x7f96f00b1e80 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.521+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e40334c0 con 0x7f96f00b1e80 2024-07-26T06:15:04.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.521+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f015fd80 con 0x7f96f00b1e80 2024-07-26T06:15:04.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.521+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f0160340 con 0x7f96f00b1e80 2024-07-26T06:15:04.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.522+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f00cf860 con 0x7f96f00b1e80 2024-07-26T06:15:04.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.523+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f96e4033660 con 0x7f96f00b1e80 2024-07-26T06:15:04.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.524+0000 7f96ed600640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f96d0076480 0x7f96d0078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:04.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.524+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f96e40b8ce0 con 0x7f96f00b1e80 2024-07-26T06:15:04.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.524+0000 7f96f4c00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f96d0076480 0x7f96d0078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:04.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.526+0000 7f96f4c00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f96d0076480 0x7f96d0078940 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f96f01f7490 tx=0x7f96dc007a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:04.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.528+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f96f00cf860 con 0x7f96f00b1e80 2024-07-26T06:15:04.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:04.717+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f96f00ce2f0 con 0x7f96f00b1e80 2024-07-26T06:15:05.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.375+0000 7f96ed600640 1 -- 172.21.15.163:0/662517133 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) ==== 138+0+0 (secure 0 0 0) 0x7f96e403f030 con 0x7f96f00b1e80 2024-07-26T06:15:05.376 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:15:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f96d0076480 msgr2=0x7f96d0078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f96d0076480 0x7f96d0078940 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f96f01f7490 tx=0x7f96dc007a30 comp rx=0 tx=0).stop 2024-07-26T06:15:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 msgr2=0x7f96f00cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 0x7f96f00cced0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009950 tx=0x7f96e4033d60 comp rx=0 tx=0).stop 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 shutdown_connections 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f96d0076480 0x7f96d0078940 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f015d5a0 0x7f96f00c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f00ce830 0x7f96f01f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 --2- 172.21.15.163:0/662517133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96f00b1e80 0x7f96f00cced0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 >> 172.21.15.163:0/662517133 conn(0x7f96f00cfda0 msgr2=0x7f96f015a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.379+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 shutdown_connections 2024-07-26T06:15:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.380+0000 7f96f6a00640 1 -- 172.21.15.163:0/662517133 wait complete. 2024-07-26T06:15:05.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:15:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.620+0000 7faa61000640 1 -- 172.21.15.163:0/3954444125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c0ce4a0 msgr2=0x7faa5c071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.620+0000 7faa61000640 1 --2- 172.21.15.163:0/3954444125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c0ce4a0 0x7faa5c071530 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7faa48009830 tx=0x7faa4802a5c0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.621+0000 7faa61000640 1 -- 172.21.15.163:0/3954444125 shutdown_connections 2024-07-26T06:15:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.621+0000 7faa61000640 1 --2- 172.21.15.163:0/3954444125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c0ce4a0 0x7faa5c071530 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.621+0000 7faa61000640 1 -- 172.21.15.163:0/3954444125 >> 172.21.15.163:0/3954444125 conn(0x7faa5c0b0e30 msgr2=0x7faa5c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.621+0000 7faa61000640 1 -- 172.21.15.163:0/3954444125 shutdown_connections 2024-07-26T06:15:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.621+0000 7faa61000640 1 -- 172.21.15.163:0/3954444125 wait complete. 2024-07-26T06:15:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.622+0000 7faa61000640 1 Processor -- start 2024-07-26T06:15:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 -- start start 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 0x7faa5c1e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa5c0ce4a0 0x7faa5c0ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa5c1556f0 0x7faa5c0acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c0b41f0 con 0x7faa5c071a70 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c0b4070 con 0x7faa5c0ce4a0 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.623+0000 7faa61000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa5c0b4370 con 0x7faa5c1556f0 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 0x7faa5c1e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 0x7faa5c1e7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55296/0 (socket says 172.21.15.163:55296) 2024-07-26T06:15:05.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 -- 172.21.15.163:0/2422444607 learned_addr learned my addr 172.21.15.163:0/2422444607 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 -- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa5c1556f0 msgr2=0x7faa5c0acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5be00640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa5c1556f0 0x7faa5c0acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5aa00640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa5c0ce4a0 0x7faa5c0ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa5c1556f0 0x7faa5c0acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 -- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa5c0ce4a0 msgr2=0x7faa5c0ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa5c0ce4a0 0x7faa5c0ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:05.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.624+0000 7faa5b400640 1 -- 172.21.15.163:0/2422444607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5c0cd210 con 0x7faa5c071a70 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.625+0000 7faa5b400640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 0x7faa5c1e7f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00c3a0 tx=0x7faa4c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.625+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c017070 con 0x7faa5c071a70 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.625+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa4c0053c0 con 0x7faa5c071a70 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.625+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4c003850 con 0x7faa5c071a70 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.625+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5c0cd4f0 con 0x7faa5c071a70 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.625+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5c0cdab0 con 0x7faa5c071a70 2024-07-26T06:15:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.626+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5c071530 con 0x7faa5c071a70 2024-07-26T06:15:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.628+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7faa4c01a070 con 0x7faa5c071a70 2024-07-26T06:15:05.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.629+0000 7faa53400640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faa3c076480 0x7faa3c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:05.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.629+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5227+0+0 (secure 0 0 0) 0x7faa4c097180 con 0x7faa5c071a70 2024-07-26T06:15:05.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.630+0000 7faa5aa00640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faa3c076480 0x7faa3c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.631+0000 7faa5aa00640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faa3c076480 0x7faa3c078940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faa48034000 tx=0x7faa48004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.632+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faa4c021020 con 0x7faa5c071a70 2024-07-26T06:15:05.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:05.836+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7faa5c0b34f0 con 0x7faa5c071a70 2024-07-26T06:15:06.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.390+0000 7faa53400640 1 -- 172.21.15.163:0/2422444607 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) ==== 170+0+0 (secure 0 0 0) 0x7faa4c060a20 con 0x7faa5c071a70 2024-07-26T06:15:06.391 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faa3c076480 msgr2=0x7faa3c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faa3c076480 0x7faa3c078940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faa48034000 tx=0x7faa48004730 comp rx=0 tx=0).stop 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 msgr2=0x7faa5c1e7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 0x7faa5c1e7f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00c3a0 tx=0x7faa4c00c860 comp rx=0 tx=0).stop 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 shutdown_connections 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7faa3c076480 0x7faa3c078940 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa5c1556f0 0x7faa5c0acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:06.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa5c0ce4a0 0x7faa5c0ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:06.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 --2- 172.21.15.163:0/2422444607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa5c071a70 0x7faa5c1e7f20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:06.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.394+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 >> 172.21.15.163:0/2422444607 conn(0x7faa5c0b0e30 msgr2=0x7faa5c15dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:06.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.395+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 shutdown_connections 2024-07-26T06:15:06.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.395+0000 7faa61000640 1 -- 172.21.15.163:0/2422444607 wait complete. 2024-07-26T06:15:06.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:15:06.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04dd400640 1 -- 172.21.15.163:0/3880079699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04bc003a00 msgr2=0x7f04bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:06.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04dd400640 1 --2- 172.21.15.163:0/3880079699 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04bc003a00 0x7f04bc003e20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f04c8001eb0 tx=0x7f04bc005830 comp rx=0 tx=0).stop 2024-07-26T06:15:06.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04d7200640 1 -- 172.21.15.163:0/3880079699 reap_dead start 2024-07-26T06:15:06.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04dd400640 1 -- 172.21.15.163:0/3880079699 shutdown_connections 2024-07-26T06:15:06.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04dd400640 1 -- 172.21.15.163:0/3880079699 >> 172.21.15.163:0/3880079699 conn(0x7f04d806d420 msgr2=0x7f04d806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:06.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04dd400640 1 -- 172.21.15.163:0/3880079699 shutdown_connections 2024-07-26T06:15:06.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.615+0000 7f04dd400640 1 -- 172.21.15.163:0/3880079699 wait complete. 2024-07-26T06:15:06.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.617+0000 7f04dd400640 1 Processor -- start 2024-07-26T06:15:06.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.617+0000 7f04dd400640 1 -- start start 2024-07-26T06:15:06.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04dd400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc003a00 0x7f04d8073c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:06.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04dd400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 0x7f04d813d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:06.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04dd400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04d8158980 0x7f04d8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04dd400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04d80ce180 con 0x7f04d813dfc0 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04dd400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f04d8161270 con 0x7f04d8158980 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04dd400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f04d80ce300 con 0x7f04bc003a00 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04cde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 0x7f04d813d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04cde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 0x7f04d813d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55316/0 (socket says 172.21.15.163:55316) 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04cde00640 1 -- 172.21.15.163:0/81687442 learned_addr learned my addr 172.21.15.163:0/81687442 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04d6800640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc003a00 0x7f04d8073c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.618+0000 7f04d7200640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04d8158980 0x7f04d8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cde00640 1 -- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc003a00 msgr2=0x7f04d8073c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cde00640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc003a00 0x7f04d8073c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cde00640 1 -- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04d8158980 msgr2=0x7f04d8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cde00640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04d8158980 0x7f04d8158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cde00640 1 -- 172.21.15.163:0/81687442 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d80741d0 con 0x7f04d813dfc0 2024-07-26T06:15:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04d7200640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04d8158980 0x7f04d8158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cde00640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 0x7f04d813d850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e9f0 tx=0x7f04c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c000cc00 con 0x7f04d813dfc0 2024-07-26T06:15:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04c000ce90 con 0x7f04d813dfc0 2024-07-26T06:15:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0005280 con 0x7f04d813dfc0 2024-07-26T06:15:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d813dd90 con 0x7f04d813dfc0 2024-07-26T06:15:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.619+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d8073890 con 0x7f04d813dfc0 2024-07-26T06:15:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.620+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d8073a90 con 0x7f04d813dfc0 2024-07-26T06:15:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.622+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f04c0005420 con 0x7f04d813dfc0 2024-07-26T06:15:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.623+0000 7f04cf400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f04b40764b0 0x7f04b4078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.623+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5609+0+0 (secure 0 0 0) 0x7f04c0098640 con 0x7f04d813dfc0 2024-07-26T06:15:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.624+0000 7f04d6800640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f04b40764b0 0x7f04b4078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:06.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.625+0000 7f04d6800640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f04b40764b0 0x7f04b4078970 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f04c8033000 tx=0x7f04c802d000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:06.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.626+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f04d8073a90 con 0x7f04d813dfc0 2024-07-26T06:15:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:06.866+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f04d806c360 con 0x7f04d813dfc0 2024-07-26T06:15:07.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.419+0000 7f04cf400640 1 -- 172.21.15.163:0/81687442 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) ==== 172+0+0 (secure 0 0 0) 0x7f04c0016030 con 0x7f04d813dfc0 2024-07-26T06:15:07.423 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 6 and data pool 7 2024-07-26T06:15:07.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.425+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f04b40764b0 msgr2=0x7f04b4078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:07.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.425+0000 7f04dd400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f04b40764b0 0x7f04b4078970 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f04c8033000 tx=0x7f04c802d000 comp rx=0 tx=0).stop 2024-07-26T06:15:07.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.425+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 msgr2=0x7f04d813d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:07.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.425+0000 7f04dd400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 0x7f04d813d850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e9f0 tx=0x7f04c000eeb0 comp rx=0 tx=0).stop 2024-07-26T06:15:07.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.426+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 shutdown_connections 2024-07-26T06:15:07.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.426+0000 7f04dd400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f04b40764b0 0x7f04b4078970 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.426+0000 7f04dd400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04d8158980 0x7f04d8158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.426+0000 7f04dd400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04d813dfc0 0x7f04d813d850 secure :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e9f0 tx=0x7f04c000eeb0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.426+0000 7f04dd400640 1 --2- 172.21.15.163:0/81687442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc003a00 0x7f04d8073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.426+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 >> 172.21.15.163:0/81687442 conn(0x7f04d806d420 msgr2=0x7f04d80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:09.500 INFO:tasks.ceph.mds.b.smithi164.stderr:2024-07-26T06:15:09.350+0000 7fdcf2e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:15:09.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.427+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 shutdown_connections 2024-07-26T06:15:09.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:07.427+0000 7f04dd400640 1 -- 172.21.15.163:0/81687442 wait complete. 2024-07-26T06:15:09.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:15:09.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 -- 172.21.15.163:0/2192295585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc650004800 msgr2=0x7fc650004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:09.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 --2- 172.21.15.163:0/2192295585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc650004800 0x7fc650004c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc658001690 tx=0x7fc650005830 comp rx=0 tx=0).stop 2024-07-26T06:15:09.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 -- 172.21.15.163:0/2192295585 shutdown_connections 2024-07-26T06:15:09.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 --2- 172.21.15.163:0/2192295585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc650004800 0x7fc650004c80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 -- 172.21.15.163:0/2192295585 >> 172.21.15.163:0/2192295585 conn(0x7fc66c0b0e30 msgr2=0x7fc66c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:09.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 -- 172.21.15.163:0/2192295585 shutdown_connections 2024-07-26T06:15:09.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.721+0000 7fc66b200640 1 -- 172.21.15.163:0/2192295585 wait complete. 2024-07-26T06:15:09.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.723+0000 7fc66b200640 1 Processor -- start 2024-07-26T06:15:09.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.723+0000 7fc66b200640 1 -- start start 2024-07-26T06:15:09.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc650004800 0x7fc66c0a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:09.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66c15c3b0 0x7fc66c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 0x7fc66c1f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c0718c0 con 0x7fc66c1f4860 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c071740 con 0x7fc650004800 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc66c071a40 con 0x7fc66c15c3b0 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 0x7fc66c1f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 0x7fc66c1f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55338/0 (socket says 172.21.15.163:55338) 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.724+0000 7fc66a800640 1 -- 172.21.15.163:0/3330544843 learned_addr learned my addr 172.21.15.163:0/3330544843 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc669400640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66c15c3b0 0x7fc66c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:09.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66a800640 1 -- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66c15c3b0 msgr2=0x7fc66c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66a800640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66c15c3b0 0x7fc66c15c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66a800640 1 -- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc650004800 msgr2=0x7fc66c0a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66a800640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc650004800 0x7fc66c0a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66a800640 1 -- 172.21.15.163:0/3330544843 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66c0a9c00 con 0x7fc66c1f4860 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66a800640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 0x7fc66c1f4ce0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc664009670 tx=0x7fc664009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664004b20 con 0x7fc66c1f4860 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc664004dd0 con 0x7fc66c1f4860 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66400eb30 con 0x7fc66c1f4860 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.725+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66c0a9ee0 con 0x7fc66c1f4860 2024-07-26T06:15:09.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.726+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66c1f5a40 con 0x7fc66c1f4860 2024-07-26T06:15:09.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.727+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66c0cdf50 con 0x7fc66c1f4860 2024-07-26T06:15:09.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.728+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fc66401a020 con 0x7fc66c1f4860 2024-07-26T06:15:09.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.729+0000 7fc662000640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc64c0764b0 0x7fc64c078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:09.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.729+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc664097b80 con 0x7fc66c1f4860 2024-07-26T06:15:09.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.729+0000 7fc669e00640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc64c0764b0 0x7fc64c078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:09.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.731+0000 7fc669e00640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc64c0764b0 0x7fc64c078970 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc65c001220 tx=0x7fc65c0015c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:09.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.733+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc6640610b0 con 0x7fc66c1f4860 2024-07-26T06:15:09.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.942+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc66c0b34f0 con 0x7fc66c1f4860 2024-07-26T06:15:09.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.943+0000 7fc662000640 1 -- 172.21.15.163:0/3330544843 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16637 (secure 0 0 0) 0x7fc66408ee20 con 0x7fc66c1f4860 2024-07-26T06:15:09.945 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:09.945 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":45,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:15:08.417060+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:15:04.734772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-07-26T06:15:05.851248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc64c0764b0 msgr2=0x7fc64c078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc64c0764b0 0x7fc64c078970 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc65c001220 tx=0x7fc65c0015c0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 msgr2=0x7fc66c1f4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 0x7fc66c1f4ce0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc664009670 tx=0x7fc664009b30 comp rx=0 tx=0).stop 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 shutdown_connections 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc64c0764b0 0x7fc64c078970 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc66c1f4860 0x7fc66c1f4ce0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66c15c3b0 0x7fc66c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 --2- 172.21.15.163:0/3330544843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc650004800 0x7fc66c0a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:09.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.946+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 >> 172.21.15.163:0/3330544843 conn(0x7fc66c0b0e30 msgr2=0x7fc66c15b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:09.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.947+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 shutdown_connections 2024-07-26T06:15:09.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:09.947+0000 7fc66b200640 1 -- 172.21.15.163:0/3330544843 wait complete. 2024-07-26T06:15:09.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:15:10.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd39d000640 1 -- 172.21.15.163:0/1588988986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd37c005160 msgr2=0x7fd37c005910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd39d000640 1 --2- 172.21.15.163:0/1588988986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd37c005160 0x7fd37c005910 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd384009b20 tx=0x7fd37c006940 comp rx=0 tx=0).stop 2024-07-26T06:15:10.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd397e00640 1 -- 172.21.15.163:0/1588988986 reap_dead start 2024-07-26T06:15:10.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd39d000640 1 -- 172.21.15.163:0/1588988986 shutdown_connections 2024-07-26T06:15:10.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd39d000640 1 -- 172.21.15.163:0/1588988986 >> 172.21.15.163:0/1588988986 conn(0x7fd3980b0e30 msgr2=0x7fd3980b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:10.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd39d000640 1 -- 172.21.15.163:0/1588988986 shutdown_connections 2024-07-26T06:15:10.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.161+0000 7fd39d000640 1 -- 172.21.15.163:0/1588988986 wait complete. 2024-07-26T06:15:10.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.163+0000 7fd39d000640 1 Processor -- start 2024-07-26T06:15:10.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.163+0000 7fd39d000640 1 -- start start 2024-07-26T06:15:10.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd39d000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd37c005160 0x7fd3980742f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd39d000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39815a7c0 0x7fd3981e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd39d000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 0x7fd3981e77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd39d000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd3980b2d80 con 0x7fd39815ac40 2024-07-26T06:15:10.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd39d000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd39806f500 con 0x7fd37c005160 2024-07-26T06:15:10.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd39d000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd3980b2f00 con 0x7fd39815a7c0 2024-07-26T06:15:10.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd397e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 0x7fd3981e77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:10.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd397e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 0x7fd3981e77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55354/0 (socket says 172.21.15.163:55354) 2024-07-26T06:15:10.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.164+0000 7fd397e00640 1 -- 172.21.15.163:0/2619239643 learned_addr learned my addr 172.21.15.163:0/2619239643 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd397e00640 1 -- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39815a7c0 msgr2=0x7fd3981e7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd397e00640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39815a7c0 0x7fd3981e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd397e00640 1 -- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd37c005160 msgr2=0x7fd3980742f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd397e00640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd37c005160 0x7fd3980742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd397e00640 1 -- 172.21.15.163:0/2619239643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39813e860 con 0x7fd39815ac40 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd397e00640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 0x7fd3981e77b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd398068e30 tx=0x7fd39000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:10.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390017070 con 0x7fd39815ac40 2024-07-26T06:15:10.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3900070d0 con 0x7fd39815ac40 2024-07-26T06:15:10.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39815b210 con 0x7fd39815ac40 2024-07-26T06:15:10.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.165+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390007520 con 0x7fd39815ac40 2024-07-26T06:15:10.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.166+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd398151ef0 con 0x7fd39815ac40 2024-07-26T06:15:10.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.167+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3980cd530 con 0x7fd39815ac40 2024-07-26T06:15:10.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.168+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fd39001a070 con 0x7fd39815ac40 2024-07-26T06:15:10.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.169+0000 7fd38f400640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd378076480 0x7fd378078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.169+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd390097930 con 0x7fd39815ac40 2024-07-26T06:15:10.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.169+0000 7fd397400640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd378076480 0x7fd378078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:10.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.170+0000 7fd397400640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd378076480 0x7fd378078940 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd388001540 tx=0x7fd3880079b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:10.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.173+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd390060f70 con 0x7fd39815ac40 2024-07-26T06:15:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.366+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd398068f00 con 0x7fd39815ac40 2024-07-26T06:15:10.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.368+0000 7fd38f400640 1 -- 172.21.15.163:0/2619239643 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16637 (secure 0 0 0) 0x7fd390060870 con 0x7fd39815ac40 2024-07-26T06:15:10.369 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:10.369 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":45,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:15:08.417060+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:15:04.734772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-07-26T06:15:05.851248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:15:10.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd378076480 msgr2=0x7fd378078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd378076480 0x7fd378078940 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd388001540 tx=0x7fd3880079b0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 msgr2=0x7fd3981e77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 0x7fd3981e77b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd398068e30 tx=0x7fd39000a430 comp rx=0 tx=0).stop 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 shutdown_connections 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd378076480 0x7fd378078940 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd39815ac40 0x7fd3981e77b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39815a7c0 0x7fd3981e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 --2- 172.21.15.163:0/2619239643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd37c005160 0x7fd3980742f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 >> 172.21.15.163:0/2619239643 conn(0x7fd3980b0e30 msgr2=0x7fd3980b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 shutdown_connections 2024-07-26T06:15:10.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.371+0000 7fd39d000640 1 -- 172.21.15.163:0/2619239643 wait complete. 2024-07-26T06:15:10.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:15:10.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a7200640 1 -- 172.21.15.163:0/1797365364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb68c004d40 msgr2=0x7fb68c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a7200640 1 --2- 172.21.15.163:0/1797365364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb68c004d40 0x7fb68c0051c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8068e30 tx=0x7fb68c005830 comp rx=0 tx=0).stop 2024-07-26T06:15:10.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a6800640 1 -- 172.21.15.163:0/1797365364 reap_dead start 2024-07-26T06:15:10.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a7200640 1 -- 172.21.15.163:0/1797365364 shutdown_connections 2024-07-26T06:15:10.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a7200640 1 -- 172.21.15.163:0/1797365364 >> 172.21.15.163:0/1797365364 conn(0x7fb6a80b0e30 msgr2=0x7fb6a80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:10.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a7200640 1 -- 172.21.15.163:0/1797365364 shutdown_connections 2024-07-26T06:15:10.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.598+0000 7fb6a7200640 1 -- 172.21.15.163:0/1797365364 wait complete. 2024-07-26T06:15:10.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.600+0000 7fb6a7200640 1 Processor -- start 2024-07-26T06:15:10.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.600+0000 7fb6a7200640 1 -- start start 2024-07-26T06:15:10.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.601+0000 7fb6a7200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb68c004d40 0x7fb6a81478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.601+0000 7fb6a7200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 0x7fb6a8074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.601+0000 7fb6a7200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6a8071860 0x7fb6a8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.601+0000 7fb6a7200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a80b2d80 con 0x7fb6a8073d80 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.601+0000 7fb6a7200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a806f500 con 0x7fb68c004d40 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.601+0000 7fb6a7200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a80b2f00 con 0x7fb6a8071860 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 0x7fb6a8074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 0x7fb6a8074200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55368/0 (socket says 172.21.15.163:55368) 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 -- 172.21.15.163:0/931538137 learned_addr learned my addr 172.21.15.163:0/931538137 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5e00640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb68c004d40 0x7fb6a81478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a6800640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6a8071860 0x7fb6a8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 -- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6a8071860 msgr2=0x7fb6a8071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6a8071860 0x7fb6a8071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 -- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb68c004d40 msgr2=0x7fb6a81478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:10.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb68c004d40 0x7fb6a81478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:10.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a5400640 1 -- 172.21.15.163:0/931538137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a8072550 con 0x7fb6a8073d80 2024-07-26T06:15:10.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.602+0000 7fb6a6800640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6a8071860 0x7fb6a8071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:10.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.603+0000 7fb6a5400640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 0x7fb6a8074200 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb69400e9e0 tx=0x7fb69400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.603+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694004260 con 0x7fb6a8073d80 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.603+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb694004400 con 0x7fb6a8073d80 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.603+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a8072830 con 0x7fb6a8073d80 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.603+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69400cbe0 con 0x7fb6a8073d80 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.603+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a8072df0 con 0x7fb6a8073d80 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.604+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a80cd530 con 0x7fb6a8073d80 2024-07-26T06:15:10.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.605+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fb6940035f0 con 0x7fb6a8073d80 2024-07-26T06:15:10.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.607+0000 7fb69e000640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6880764b0 0x7fb688078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:10.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.607+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fb6940983b0 con 0x7fb6a8073d80 2024-07-26T06:15:10.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.608+0000 7fb6a5e00640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6880764b0 0x7fb688078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:10.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.609+0000 7fb6a5e00640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6880764b0 0x7fb688078970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb698001220 tx=0x7fb698009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:10.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.611+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb6940618e0 con 0x7fb6a8073d80 2024-07-26T06:15:10.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:10.857+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb6a80b3d00 con 0x7fb6a8073d80 2024-07-26T06:15:11.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.423+0000 7fb69e000640 1 -- 172.21.15.163:0/931538137 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7fb6940611e0 con 0x7fb6a8073d80 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6880764b0 msgr2=0x7fb688078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6880764b0 0x7fb688078970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb698001220 tx=0x7fb698009040 comp rx=0 tx=0).stop 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 msgr2=0x7fb6a8074200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 0x7fb6a8074200 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb69400e9e0 tx=0x7fb69400eea0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 shutdown_connections 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb6880764b0 0x7fb688078970 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6a8071860 0x7fb6a8071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6a8073d80 0x7fb6a8074200 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 --2- 172.21.15.163:0/931538137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb68c004d40 0x7fb6a81478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.426+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 >> 172.21.15.163:0/931538137 conn(0x7fb6a80b0e30 msgr2=0x7fb6a8156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:11.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.427+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 shutdown_connections 2024-07-26T06:15:11.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.427+0000 7fb6a7200640 1 -- 172.21.15.163:0/931538137 wait complete. 2024-07-26T06:15:11.451 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:15:11.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:15:11.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 -- 172.21.15.163:0/1215564555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c06f2f0 msgr2=0x7fcd3c0b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:11.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 --2- 172.21.15.163:0/1215564555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c06f2f0 0x7fcd3c0b4c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009830 tx=0x7fcd2802a510 comp rx=0 tx=0).stop 2024-07-26T06:15:11.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 -- 172.21.15.163:0/1215564555 shutdown_connections 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 --2- 172.21.15.163:0/1215564555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c06f2f0 0x7fcd3c0b4c80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 --2- 172.21.15.163:0/1215564555 >> v2:172.21.15.164:3300/0 conn(0x7fcd3c0ccdc0 0x7fcd3c1590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 -- 172.21.15.163:0/1215564555 >> 172.21.15.163:0/1215564555 conn(0x7fcd3c0b0e30 msgr2=0x7fcd3c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 -- 172.21.15.163:0/1215564555 shutdown_connections 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.664+0000 7fcd40c00640 1 -- 172.21.15.163:0/1215564555 wait complete. 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.666+0000 7fcd40c00640 1 Processor -- start 2024-07-26T06:15:11.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.667+0000 7fcd40c00640 1 -- start start 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd40c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd3c06f2f0 0x7fcd3c154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd40c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 0x7fcd3c1f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd40c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c15ccb0 0x7fcd3c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd40c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c15d9f0 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd40c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c15d870 con 0x7fcd3c06f2f0 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd40c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcd3c0cd3a0 con 0x7fcd3c15ccb0 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd3aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 0x7fcd3c1f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:11.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd3aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 0x7fcd3c1f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55376/0 (socket says 172.21.15.163:55376) 2024-07-26T06:15:11.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.668+0000 7fcd3aa00640 1 -- 172.21.15.163:0/4215126479 learned_addr learned my addr 172.21.15.163:0/4215126479 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3be00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c15ccb0 0x7fcd3c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3aa00640 1 -- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c15ccb0 msgr2=0x7fcd3c15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3aa00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c15ccb0 0x7fcd3c15a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3aa00640 1 -- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd3c06f2f0 msgr2=0x7fcd3c154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3b400640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd3c06f2f0 0x7fcd3c154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3aa00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd3c06f2f0 0x7fcd3c154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3aa00640 1 -- 172.21.15.163:0/4215126479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3c15b040 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.669+0000 7fcd3aa00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 0x7fcd3c1f3300 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009950 tx=0x7fcd28032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.670+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28040070 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.670+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd28005310 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.670+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3c15b320 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.670+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28005c90 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.670+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd3c15b8e0 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.671+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd3c1f3840 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.672+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fcd28039040 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.674+0000 7fcd33400640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcd1c076480 0x7fcd1c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:11.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.674+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fcd280b8e70 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.674+0000 7fcd3b400640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcd1c076480 0x7fcd1c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:11.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.676+0000 7fcd3b400640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcd1c076480 0x7fcd1c078940 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00ab10 tx=0x7fcd2c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:11.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.678+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcd280823a0 con 0x7fcd3c0ccdc0 2024-07-26T06:15:11.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:11.904+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fcd3c0b4c80 con 0x7fcd3c0ccdc0 2024-07-26T06:15:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.433+0000 7fcd33400640 1 -- 172.21.15.163:0/4215126479 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7fcd28081ca0 con 0x7fcd3c0ccdc0 2024-07-26T06:15:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcd1c076480 msgr2=0x7fcd1c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcd1c076480 0x7fcd1c078940 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00ab10 tx=0x7fcd2c005f90 comp rx=0 tx=0).stop 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 msgr2=0x7fcd3c1f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 0x7fcd3c1f3300 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009950 tx=0x7fcd28032c10 comp rx=0 tx=0).stop 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 shutdown_connections 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcd1c076480 0x7fcd1c078940 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd3c15ccb0 0x7fcd3c15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcd3c0ccdc0 0x7fcd3c1f3300 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 --2- 172.21.15.163:0/4215126479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd3c06f2f0 0x7fcd3c154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 >> 172.21.15.163:0/4215126479 conn(0x7fcd3c0b0e30 msgr2=0x7fcd3c14be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.437+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 shutdown_connections 2024-07-26T06:15:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.438+0000 7fcd40c00640 1 -- 172.21.15.163:0/4215126479 wait complete. 2024-07-26T06:15:12.461 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:15:12.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:12.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 -- 172.21.15.163:0/264670629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e500b6500 msgr2=0x7f7e500b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 --2- 172.21.15.163:0/264670629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e500b6500 0x7f7e500b6910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7e500b61e0 tx=0x7f7e4802a680 comp rx=0 tx=0).stop 2024-07-26T06:15:12.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 -- 172.21.15.163:0/264670629 shutdown_connections 2024-07-26T06:15:12.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 --2- 172.21.15.163:0/264670629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e500b6500 0x7f7e500b6910 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 --2- 172.21.15.163:0/264670629 >> v2:172.21.15.164:3300/0 conn(0x7f7e500cfcc0 0x7f7e500d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 -- 172.21.15.163:0/264670629 >> 172.21.15.163:0/264670629 conn(0x7f7e5006d880 msgr2=0x7f7e5006dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:12.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 -- 172.21.15.163:0/264670629 shutdown_connections 2024-07-26T06:15:12.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.664+0000 7f7e4f200640 1 -- 172.21.15.163:0/264670629 wait complete. 2024-07-26T06:15:12.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.666+0000 7f7e4f200640 1 Processor -- start 2024-07-26T06:15:12.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.666+0000 7f7e4f200640 1 -- start start 2024-07-26T06:15:12.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e500b6500 0x7f7e500cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:12.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500cfcc0 0x7f7e500ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:12.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 0x7f7e500a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:12.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7e500b6e50 con 0x7f7e5015d3f0 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7e500b6fd0 con 0x7f7e500b6500 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7e500b7150 con 0x7f7e500cfcc0 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 0x7f7e500a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 0x7f7e500a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55404/0 (socket says 172.21.15.163:55404) 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 -- 172.21.15.163:0/1092637081 learned_addr learned my addr 172.21.15.163:0/1092637081 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 -- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500cfcc0 msgr2=0x7f7e500ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500cfcc0 0x7f7e500ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 -- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e500b6500 msgr2=0x7f7e500cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e500b6500 0x7f7e500cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.667+0000 7f7e4e800640 1 -- 172.21.15.163:0/1092637081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e50154100 con 0x7f7e5015d3f0 2024-07-26T06:15:12.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.668+0000 7f7e4e800640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 0x7f7e500a9e60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009230 tx=0x7f7e48003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:12.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.668+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4803f070 con 0x7f7e5015d3f0 2024-07-26T06:15:12.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.668+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e48004890 con 0x7f7e5015d3f0 2024-07-26T06:15:12.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.668+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48035030 con 0x7f7e5015d3f0 2024-07-26T06:15:12.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.668+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e501543e0 con 0x7f7e5015d3f0 2024-07-26T06:15:12.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.668+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e500b08a0 con 0x7f7e5015d3f0 2024-07-26T06:15:12.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.669+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e500b7600 con 0x7f7e5015d3f0 2024-07-26T06:15:12.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.670+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f7e48042070 con 0x7f7e5015d3f0 2024-07-26T06:15:12.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.671+0000 7f7e46a00640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7e2c0764b0 0x7f7e2c078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:12.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.671+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f7e480b8a30 con 0x7f7e5015d3f0 2024-07-26T06:15:12.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.672+0000 7f7e4de00640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7e2c0764b0 0x7f7e2c078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:12.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.673+0000 7f7e4de00640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7e2c0764b0 0x7f7e2c078970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7e400012e0 tx=0x7f7e40007d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:12.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.676+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e480820c0 con 0x7f7e5015d3f0 2024-07-26T06:15:12.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.919+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e500d3080 con 0x7f7e5015d3f0 2024-07-26T06:15:12.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.921+0000 7f7e46a00640 1 -- 172.21.15.163:0/1092637081 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f7e4808a0b0 con 0x7f7e5015d3f0 2024-07-26T06:15:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:12.922 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":25,"btime":"2024-07-26T06:15:12:424563+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:12.922 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 25 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.923+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7e2c0764b0 msgr2=0x7f7e2c078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.923+0000 7f7e4f200640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7e2c0764b0 0x7f7e2c078970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7e400012e0 tx=0x7f7e40007d30 comp rx=0 tx=0).stop 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.923+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 msgr2=0x7f7e500a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.923+0000 7f7e4f200640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 0x7f7e500a9e60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009230 tx=0x7f7e48003960 comp rx=0 tx=0).stop 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 shutdown_connections 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7e2c0764b0 0x7f7e2c078970 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e5015d3f0 0x7f7e500a9e60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500cfcc0 0x7f7e500ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 --2- 172.21.15.163:0/1092637081 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e500b6500 0x7f7e500cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:12.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 >> 172.21.15.163:0/1092637081 conn(0x7f7e5006d880 msgr2=0x7f7e500b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:12.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 shutdown_connections 2024-07-26T06:15:12.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:12.924+0000 7f7e4f200640 1 -- 172.21.15.163:0/1092637081 wait complete. 2024-07-26T06:15:12.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:15:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 -- 172.21.15.163:0/1702194817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8080ce4a0 msgr2=0x7fc808071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 --2- 172.21.15.163:0/1702194817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8080ce4a0 0x7fc808071530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009830 tx=0x7fc7f402a510 comp rx=0 tx=0).stop 2024-07-26T06:15:13.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 -- 172.21.15.163:0/1702194817 shutdown_connections 2024-07-26T06:15:13.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 --2- 172.21.15.163:0/1702194817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc808071a70 0x7fc8080b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 --2- 172.21.15.163:0/1702194817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8080ce4a0 0x7fc808071530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 -- 172.21.15.163:0/1702194817 >> 172.21.15.163:0/1702194817 conn(0x7fc8080b0e30 msgr2=0x7fc8080b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:13.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 -- 172.21.15.163:0/1702194817 shutdown_connections 2024-07-26T06:15:13.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.138+0000 7fc80f000640 1 -- 172.21.15.163:0/1702194817 wait complete. 2024-07-26T06:15:13.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.140+0000 7fc80f000640 1 Processor -- start 2024-07-26T06:15:13.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.140+0000 7fc80f000640 1 -- start start 2024-07-26T06:15:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80f000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 0x7fc808144a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80f000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8080ce4a0 0x7fc808153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80f000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8081556f0 0x7fc808158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80f000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc8080b3fe0 con 0x7fc808071a70 2024-07-26T06:15:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80f000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc8080b3e60 con 0x7fc8081556f0 2024-07-26T06:15:13.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80f000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc8080b4160 con 0x7fc8080ce4a0 2024-07-26T06:15:13.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.141+0000 7fc80dc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 0x7fc808144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 0x7fc808144a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55412/0 (socket says 172.21.15.163:55412) 2024-07-26T06:15:13.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 -- 172.21.15.163:0/1346021234 learned_addr learned my addr 172.21.15.163:0/1346021234 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80d200640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8080ce4a0 0x7fc808153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 -- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8080ce4a0 msgr2=0x7fc808153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8080ce4a0 0x7fc808153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 -- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8081556f0 msgr2=0x7fc808158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8081556f0 0x7fc808158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 -- 172.21.15.163:0/1346021234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc808153ff0 con 0x7fc808071a70 2024-07-26T06:15:13.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.142+0000 7fc80dc00640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 0x7fc808144a50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00c370 tx=0x7fc7fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.143+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc006890 con 0x7fc808071a70 2024-07-26T06:15:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.143+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7fc006b20 con 0x7fc808071a70 2024-07-26T06:15:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.143+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc007020 con 0x7fc808071a70 2024-07-26T06:15:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.143+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8081542d0 con 0x7fc808071a70 2024-07-26T06:15:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.143+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc808153b50 con 0x7fc808071a70 2024-07-26T06:15:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.144+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8080b46f0 con 0x7fc808071a70 2024-07-26T06:15:13.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.145+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fc7fc019020 con 0x7fc808071a70 2024-07-26T06:15:13.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.146+0000 7fc805600640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7e8076480 0x7fc7e8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.146+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc7fc0971d0 con 0x7fc808071a70 2024-07-26T06:15:13.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.147+0000 7fc80d200640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7e8076480 0x7fc7e8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.148+0000 7fc80d200640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7e8076480 0x7fc7e8078940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4034000 tx=0x7fc7f4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:13.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.151+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc7fc0608a0 con 0x7fc808071a70 2024-07-26T06:15:13.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.357+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8080b34f0 con 0x7fc808071a70 2024-07-26T06:15:13.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.360+0000 7fc805600640 1 -- 172.21.15.163:0/1346021234 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16637 (secure 0 0 0) 0x7fc7fc0601a0 con 0x7fc808071a70 2024-07-26T06:15:13.361 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:13.361 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":45,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:15:08.417060+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:15:04.734772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-07-26T06:15:05.851248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:15:13.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7e8076480 msgr2=0x7fc7e8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7e8076480 0x7fc7e8078940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4034000 tx=0x7fc7f4004730 comp rx=0 tx=0).stop 2024-07-26T06:15:13.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 msgr2=0x7fc808144a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 0x7fc808144a50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00c370 tx=0x7fc7fc00c830 comp rx=0 tx=0).stop 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 shutdown_connections 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc7e8076480 0x7fc7e8078940 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8081556f0 0x7fc808158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8080ce4a0 0x7fc808153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 --2- 172.21.15.163:0/1346021234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc808071a70 0x7fc808144a50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 >> 172.21.15.163:0/1346021234 conn(0x7fc8080b0e30 msgr2=0x7fc80815be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 shutdown_connections 2024-07-26T06:15:13.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.363+0000 7fc80f000640 1 -- 172.21.15.163:0/1346021234 wait complete. 2024-07-26T06:15:13.385 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:15:13.385 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:15:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.586+0000 7efe2b200640 1 -- 172.21.15.163:0/3948753742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 msgr2=0x7efe240b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.586+0000 7efe2b200640 1 --2- 172.21.15.163:0/3948753742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b64c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efe2406f160 tx=0x7efe1402a440 comp rx=0 tx=0).stop 2024-07-26T06:15:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.587+0000 7efe2b200640 1 -- 172.21.15.163:0/3948753742 shutdown_connections 2024-07-26T06:15:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.587+0000 7efe2b200640 1 --2- 172.21.15.163:0/3948753742 >> v2:172.21.15.164:3300/0 conn(0x7efe240b23d0 0x7efe240b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.587+0000 7efe2b200640 1 --2- 172.21.15.163:0/3948753742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b64c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.587+0000 7efe2b200640 1 -- 172.21.15.163:0/3948753742 >> 172.21.15.163:0/3948753742 conn(0x7efe240cfda0 msgr2=0x7efe240d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:13.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.587+0000 7efe2b200640 1 -- 172.21.15.163:0/3948753742 shutdown_connections 2024-07-26T06:15:13.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.587+0000 7efe2b200640 1 -- 172.21.15.163:0/3948753742 wait complete. 2024-07-26T06:15:13.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.589+0000 7efe2b200640 1 Processor -- start 2024-07-26T06:15:13.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.589+0000 7efe2b200640 1 -- start start 2024-07-26T06:15:13.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.590+0000 7efe2b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.590+0000 7efe2b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe240b23d0 0x7efe24156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.590+0000 7efe2b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe2415d5a0 0x7efe24156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.590+0000 7efe2b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efe240b4f20 con 0x7efe240b1e80 2024-07-26T06:15:13.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.590+0000 7efe2b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efe240b4da0 con 0x7efe240b23d0 2024-07-26T06:15:13.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.590+0000 7efe2b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efe240b50a0 con 0x7efe2415d5a0 2024-07-26T06:15:13.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55424/0 (socket says 172.21.15.163:55424) 2024-07-26T06:15:13.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 -- 172.21.15.163:0/180748775 learned_addr learned my addr 172.21.15.163:0/180748775 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:13.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29400640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe240b23d0 0x7efe24156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe2a800640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe2415d5a0 0x7efe24156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 -- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe2415d5a0 msgr2=0x7efe24156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe2415d5a0 0x7efe24156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 -- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe240b23d0 msgr2=0x7efe24156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe240b23d0 0x7efe24156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 -- 172.21.15.163:0/180748775 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe241550a0 con 0x7efe240b1e80 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe2a800640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe2415d5a0 0x7efe24156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.591+0000 7efe29e00640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b84d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efe1800b3e0 tx=0x7efe1800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.592+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18015020 con 0x7efe240b1e80 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.592+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe18011940 con 0x7efe240b1e80 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.592+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe24155380 con 0x7efe240b1e80 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.592+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18011d90 con 0x7efe240b1e80 2024-07-26T06:15:13.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.592+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe24155940 con 0x7efe240b1e80 2024-07-26T06:15:13.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.593+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe240b5570 con 0x7efe240b1e80 2024-07-26T06:15:13.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.594+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7efe18018020 con 0x7efe240b1e80 2024-07-26T06:15:13.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.596+0000 7efe1e000640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efe080764b0 0x7efe08078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:13.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.596+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7efe18097a80 con 0x7efe240b1e80 2024-07-26T06:15:13.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.596+0000 7efe29400640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efe080764b0 0x7efe08078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:13.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.598+0000 7efe29400640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efe080764b0 0x7efe08078970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efe14033000 tx=0x7efe14004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:13.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.601+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe18099e20 con 0x7efe240b1e80 2024-07-26T06:15:13.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.803+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe2415c840 con 0x7efe080764b0 2024-07-26T06:15:13.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.820+0000 7efe1e000640 1 -- 172.21.15.163:0/180748775 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7efe2415c840 con 0x7efe080764b0 2024-07-26T06:15:13.825 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:13.827 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:15:13.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efe080764b0 msgr2=0x7efe08078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efe080764b0 0x7efe08078970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efe14033000 tx=0x7efe14004730 comp rx=0 tx=0).stop 2024-07-26T06:15:13.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 msgr2=0x7efe240b84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:13.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b84d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efe1800b3e0 tx=0x7efe1800b8a0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 shutdown_connections 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efe080764b0 0x7efe08078970 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe2415d5a0 0x7efe24156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe240b23d0 0x7efe24156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 --2- 172.21.15.163:0/180748775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe240b1e80 0x7efe240b84d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 >> 172.21.15.163:0/180748775 conn(0x7efe240cfda0 msgr2=0x7efe240b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 shutdown_connections 2024-07-26T06:15:13.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:13.827+0000 7efe2b200640 1 -- 172.21.15.163:0/180748775 wait complete. 2024-07-26T06:15:13.849 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-07-26T06:15:13.678704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225616,"kb_used_data":5288,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749104,"statfs":{"total":773094113280,"available":772863082496,"internally_reserved":0,"allocated":5414912,"data_stored":2871196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.284852"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429503+0000","last_change":"2024-07-26T06:15:07.439002+0000","last_active":"2024-07-26T06:15:08.429503+0000","last_peered":"2024-07-26T06:15:08.429503+0000","last_clean":"2024-07-26T06:15:08.429503+0000","last_became_active":"2024-07-26T06:15:07.437642+0000","last_became_peered":"2024-07-26T06:15:07.437642+0000","last_unstale":"2024-07-26T06:15:08.429503+0000","last_undegraded":"2024-07-26T06:15:08.429503+0000","last_fullsized":"2024-07-26T06:15:08.429503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:37.698071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165913+0000","last_change":"2024-07-26T06:15:07.443943+0000","last_active":"2024-07-26T06:15:10.165913+0000","last_peered":"2024-07-26T06:15:10.165913+0000","last_clean":"2024-07-26T06:15:10.165913+0000","last_became_active":"2024-07-26T06:15:07.443630+0000","last_became_peered":"2024-07-26T06:15:07.443630+0000","last_unstale":"2024-07-26T06:15:10.165913+0000","last_undegraded":"2024-07-26T06:15:10.165913+0000","last_fullsized":"2024-07-26T06:15:10.165913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:04.663181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442388+0000","last_change":"2024-07-26T06:15:07.428380+0000","last_active":"2024-07-26T06:15:08.442388+0000","last_peered":"2024-07-26T06:15:08.442388+0000","last_clean":"2024-07-26T06:15:08.442388+0000","last_became_active":"2024-07-26T06:15:07.428033+0000","last_became_peered":"2024-07-26T06:15:07.428033+0000","last_unstale":"2024-07-26T06:15:08.442388+0000","last_undegraded":"2024-07-26T06:15:08.442388+0000","last_fullsized":"2024-07-26T06:15:08.442388+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:57:29.467958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427784+0000","last_change":"2024-07-26T06:15:07.446508+0000","last_active":"2024-07-26T06:15:08.427784+0000","last_peered":"2024-07-26T06:15:08.427784+0000","last_clean":"2024-07-26T06:15:08.427784+0000","last_became_active":"2024-07-26T06:15:07.446277+0000","last_became_peered":"2024-07-26T06:15:07.446277+0000","last_unstale":"2024-07-26T06:15:08.427784+0000","last_undegraded":"2024-07-26T06:15:08.427784+0000","last_fullsized":"2024-07-26T06:15:08.427784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:56.749715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165115+0000","last_change":"2024-07-26T06:15:07.442881+0000","last_active":"2024-07-26T06:15:10.165115+0000","last_peered":"2024-07-26T06:15:10.165115+0000","last_clean":"2024-07-26T06:15:10.165115+0000","last_became_active":"2024-07-26T06:15:07.442563+0000","last_became_peered":"2024-07-26T06:15:07.442563+0000","last_unstale":"2024-07-26T06:15:10.165115+0000","last_undegraded":"2024-07-26T06:15:10.165115+0000","last_fullsized":"2024-07-26T06:15:10.165115+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:38.427858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431092+0000","last_change":"2024-07-26T06:15:07.439882+0000","last_active":"2024-07-26T06:15:08.431092+0000","last_peered":"2024-07-26T06:15:08.431092+0000","last_clean":"2024-07-26T06:15:08.431092+0000","last_became_active":"2024-07-26T06:15:07.439317+0000","last_became_peered":"2024-07-26T06:15:07.439317+0000","last_unstale":"2024-07-26T06:15:08.431092+0000","last_undegraded":"2024-07-26T06:15:08.431092+0000","last_fullsized":"2024-07-26T06:15:08.431092+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:22:51.472360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428232+0000","last_change":"2024-07-26T06:15:07.441524+0000","last_active":"2024-07-26T06:15:08.428232+0000","last_peered":"2024-07-26T06:15:08.428232+0000","last_clean":"2024-07-26T06:15:08.428232+0000","last_became_active":"2024-07-26T06:15:07.441230+0000","last_became_peered":"2024-07-26T06:15:07.441230+0000","last_unstale":"2024-07-26T06:15:08.428232+0000","last_undegraded":"2024-07-26T06:15:08.428232+0000","last_fullsized":"2024-07-26T06:15:08.428232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:04.663181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427054+0000","last_change":"2024-07-26T06:15:07.456220+0000","last_active":"2024-07-26T06:15:08.427054+0000","last_peered":"2024-07-26T06:15:08.427054+0000","last_clean":"2024-07-26T06:15:08.427054+0000","last_became_active":"2024-07-26T06:15:07.455408+0000","last_became_peered":"2024-07-26T06:15:07.455408+0000","last_unstale":"2024-07-26T06:15:08.427054+0000","last_undegraded":"2024-07-26T06:15:08.427054+0000","last_fullsized":"2024-07-26T06:15:08.427054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:16:49.612701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430334+0000","last_change":"2024-07-26T06:15:07.430412+0000","last_active":"2024-07-26T06:15:08.430334+0000","last_peered":"2024-07-26T06:15:08.430334+0000","last_clean":"2024-07-26T06:15:08.430334+0000","last_became_active":"2024-07-26T06:15:07.430114+0000","last_became_peered":"2024-07-26T06:15:07.430114+0000","last_unstale":"2024-07-26T06:15:08.430334+0000","last_undegraded":"2024-07-26T06:15:08.430334+0000","last_fullsized":"2024-07-26T06:15:08.430334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:38.427858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430061+0000","last_change":"2024-07-26T06:15:07.454615+0000","last_active":"2024-07-26T06:15:08.430061+0000","last_peered":"2024-07-26T06:15:08.430061+0000","last_clean":"2024-07-26T06:15:08.430061+0000","last_became_active":"2024-07-26T06:15:07.454233+0000","last_became_peered":"2024-07-26T06:15:07.454233+0000","last_unstale":"2024-07-26T06:15:08.430061+0000","last_undegraded":"2024-07-26T06:15:08.430061+0000","last_fullsized":"2024-07-26T06:15:08.430061+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:55.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429430+0000","last_change":"2024-07-26T06:15:07.428324+0000","last_active":"2024-07-26T06:15:08.429430+0000","last_peered":"2024-07-26T06:15:08.429430+0000","last_clean":"2024-07-26T06:15:08.429430+0000","last_became_active":"2024-07-26T06:15:07.428005+0000","last_became_peered":"2024-07-26T06:15:07.428005+0000","last_unstale":"2024-07-26T06:15:08.429430+0000","last_undegraded":"2024-07-26T06:15:08.429430+0000","last_fullsized":"2024-07-26T06:15:08.429430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:49.119077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165498+0000","last_change":"2024-07-26T06:15:07.440834+0000","last_active":"2024-07-26T06:15:10.165498+0000","last_peered":"2024-07-26T06:15:10.165498+0000","last_clean":"2024-07-26T06:15:10.165498+0000","last_became_active":"2024-07-26T06:15:07.440254+0000","last_became_peered":"2024-07-26T06:15:07.440254+0000","last_unstale":"2024-07-26T06:15:10.165498+0000","last_undegraded":"2024-07-26T06:15:10.165498+0000","last_fullsized":"2024-07-26T06:15:10.165498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:44.629861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429826+0000","last_change":"2024-07-26T06:15:07.434318+0000","last_active":"2024-07-26T06:15:08.429826+0000","last_peered":"2024-07-26T06:15:08.429826+0000","last_clean":"2024-07-26T06:15:08.429826+0000","last_became_active":"2024-07-26T06:15:07.434035+0000","last_became_peered":"2024-07-26T06:15:07.434035+0000","last_unstale":"2024-07-26T06:15:08.429826+0000","last_undegraded":"2024-07-26T06:15:08.429826+0000","last_fullsized":"2024-07-26T06:15:08.429826+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:32.612822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427745+0000","last_change":"2024-07-26T06:15:07.428831+0000","last_active":"2024-07-26T06:15:08.427745+0000","last_peered":"2024-07-26T06:15:08.427745+0000","last_clean":"2024-07-26T06:15:08.427745+0000","last_became_active":"2024-07-26T06:15:07.428547+0000","last_became_peered":"2024-07-26T06:15:07.428547+0000","last_unstale":"2024-07-26T06:15:08.427745+0000","last_undegraded":"2024-07-26T06:15:08.427745+0000","last_fullsized":"2024-07-26T06:15:08.427745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:35.122591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431216+0000","last_change":"2024-07-26T06:15:07.421441+0000","last_active":"2024-07-26T06:15:08.431216+0000","last_peered":"2024-07-26T06:15:08.431216+0000","last_clean":"2024-07-26T06:15:08.431216+0000","last_became_active":"2024-07-26T06:15:07.421000+0000","last_became_peered":"2024-07-26T06:15:07.421000+0000","last_unstale":"2024-07-26T06:15:08.431216+0000","last_undegraded":"2024-07-26T06:15:08.431216+0000","last_fullsized":"2024-07-26T06:15:08.431216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:35:48.576971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063818+0000","last_change":"2024-07-26T06:15:07.447544+0000","last_active":"2024-07-26T06:15:09.063818+0000","last_peered":"2024-07-26T06:15:09.063818+0000","last_clean":"2024-07-26T06:15:09.063818+0000","last_became_active":"2024-07-26T06:15:07.447200+0000","last_became_peered":"2024-07-26T06:15:07.447200+0000","last_unstale":"2024-07-26T06:15:09.063818+0000","last_undegraded":"2024-07-26T06:15:09.063818+0000","last_fullsized":"2024-07-26T06:15:09.063818+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442761+0000","last_change":"2024-07-26T06:15:07.433146+0000","last_active":"2024-07-26T06:15:08.442761+0000","last_peered":"2024-07-26T06:15:08.442761+0000","last_clean":"2024-07-26T06:15:08.442761+0000","last_became_active":"2024-07-26T06:15:07.432583+0000","last_became_peered":"2024-07-26T06:15:07.432583+0000","last_unstale":"2024-07-26T06:15:08.442761+0000","last_undegraded":"2024-07-26T06:15:08.442761+0000","last_fullsized":"2024-07-26T06:15:08.442761+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:11.453191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442768+0000","last_change":"2024-07-26T06:15:07.431772+0000","last_active":"2024-07-26T06:15:08.442768+0000","last_peered":"2024-07-26T06:15:08.442768+0000","last_clean":"2024-07-26T06:15:08.442768+0000","last_became_active":"2024-07-26T06:15:07.430990+0000","last_became_peered":"2024-07-26T06:15:07.430990+0000","last_unstale":"2024-07-26T06:15:08.442768+0000","last_undegraded":"2024-07-26T06:15:08.442768+0000","last_fullsized":"2024-07-26T06:15:08.442768+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:19:53.598400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063547+0000","last_change":"2024-07-26T06:15:07.446491+0000","last_active":"2024-07-26T06:15:09.063547+0000","last_peered":"2024-07-26T06:15:09.063547+0000","last_clean":"2024-07-26T06:15:09.063547+0000","last_became_active":"2024-07-26T06:15:07.446167+0000","last_became_peered":"2024-07-26T06:15:07.446167+0000","last_unstale":"2024-07-26T06:15:09.063547+0000","last_undegraded":"2024-07-26T06:15:09.063547+0000","last_fullsized":"2024-07-26T06:15:09.063547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:00:47.313017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.433244+0000","last_change":"2024-07-26T06:15:07.430059+0000","last_active":"2024-07-26T06:15:08.433244+0000","last_peered":"2024-07-26T06:15:08.433244+0000","last_clean":"2024-07-26T06:15:08.433244+0000","last_became_active":"2024-07-26T06:15:07.429271+0000","last_became_peered":"2024-07-26T06:15:07.429271+0000","last_unstale":"2024-07-26T06:15:08.433244+0000","last_undegraded":"2024-07-26T06:15:08.433244+0000","last_fullsized":"2024-07-26T06:15:08.433244+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:38.518133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431704+0000","last_change":"2024-07-26T06:15:07.447086+0000","last_active":"2024-07-26T06:15:08.431704+0000","last_peered":"2024-07-26T06:15:08.431704+0000","last_clean":"2024-07-26T06:15:08.431704+0000","last_became_active":"2024-07-26T06:15:07.446851+0000","last_became_peered":"2024-07-26T06:15:07.446851+0000","last_unstale":"2024-07-26T06:15:08.431704+0000","last_undegraded":"2024-07-26T06:15:08.431704+0000","last_fullsized":"2024-07-26T06:15:08.431704+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:34:53.383778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428183+0000","last_change":"2024-07-26T06:15:07.455293+0000","last_active":"2024-07-26T06:15:08.428183+0000","last_peered":"2024-07-26T06:15:08.428183+0000","last_clean":"2024-07-26T06:15:08.428183+0000","last_became_active":"2024-07-26T06:15:07.454786+0000","last_became_peered":"2024-07-26T06:15:07.454786+0000","last_unstale":"2024-07-26T06:15:08.428183+0000","last_undegraded":"2024-07-26T06:15:08.428183+0000","last_fullsized":"2024-07-26T06:15:08.428183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:24:58.110547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063714+0000","last_change":"2024-07-26T06:15:07.448843+0000","last_active":"2024-07-26T06:15:09.063714+0000","last_peered":"2024-07-26T06:15:09.063714+0000","last_clean":"2024-07-26T06:15:09.063714+0000","last_became_active":"2024-07-26T06:15:07.448567+0000","last_became_peered":"2024-07-26T06:15:07.448567+0000","last_unstale":"2024-07-26T06:15:09.063714+0000","last_undegraded":"2024-07-26T06:15:09.063714+0000","last_fullsized":"2024-07-26T06:15:09.063714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:22.751385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063090+0000","last_change":"2024-07-26T06:15:07.444695+0000","last_active":"2024-07-26T06:15:09.063090+0000","last_peered":"2024-07-26T06:15:09.063090+0000","last_clean":"2024-07-26T06:15:09.063090+0000","last_became_active":"2024-07-26T06:15:07.444418+0000","last_became_peered":"2024-07-26T06:15:07.444418+0000","last_unstale":"2024-07-26T06:15:09.063090+0000","last_undegraded":"2024-07-26T06:15:09.063090+0000","last_fullsized":"2024-07-26T06:15:09.063090+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:20.118927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063436+0000","last_change":"2024-07-26T06:15:07.454543+0000","last_active":"2024-07-26T06:15:09.063436+0000","last_peered":"2024-07-26T06:15:09.063436+0000","last_clean":"2024-07-26T06:15:09.063436+0000","last_became_active":"2024-07-26T06:15:07.452159+0000","last_became_peered":"2024-07-26T06:15:07.452159+0000","last_unstale":"2024-07-26T06:15:09.063436+0000","last_undegraded":"2024-07-26T06:15:09.063436+0000","last_fullsized":"2024-07-26T06:15:09.063436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:35:53.361444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430097+0000","last_change":"2024-07-26T06:15:07.434897+0000","last_active":"2024-07-26T06:15:08.430097+0000","last_peered":"2024-07-26T06:15:08.430097+0000","last_clean":"2024-07-26T06:15:08.430097+0000","last_became_active":"2024-07-26T06:15:07.434456+0000","last_became_peered":"2024-07-26T06:15:07.434456+0000","last_unstale":"2024-07-26T06:15:08.430097+0000","last_undegraded":"2024-07-26T06:15:08.430097+0000","last_fullsized":"2024-07-26T06:15:08.430097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:07:44.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442528+0000","last_change":"2024-07-26T06:15:07.442852+0000","last_active":"2024-07-26T06:15:08.442528+0000","last_peered":"2024-07-26T06:15:08.442528+0000","last_clean":"2024-07-26T06:15:08.442528+0000","last_became_active":"2024-07-26T06:15:07.440917+0000","last_became_peered":"2024-07-26T06:15:07.440917+0000","last_unstale":"2024-07-26T06:15:08.442528+0000","last_undegraded":"2024-07-26T06:15:08.442528+0000","last_fullsized":"2024-07-26T06:15:08.442528+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:33.985738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165675+0000","last_change":"2024-07-26T06:15:07.440581+0000","last_active":"2024-07-26T06:15:10.165675+0000","last_peered":"2024-07-26T06:15:10.165675+0000","last_clean":"2024-07-26T06:15:10.165675+0000","last_became_active":"2024-07-26T06:15:07.439909+0000","last_became_peered":"2024-07-26T06:15:07.439909+0000","last_unstale":"2024-07-26T06:15:10.165675+0000","last_undegraded":"2024-07-26T06:15:10.165675+0000","last_fullsized":"2024-07-26T06:15:10.165675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:21:23.009970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063424+0000","last_change":"2024-07-26T06:15:07.452847+0000","last_active":"2024-07-26T06:15:09.063424+0000","last_peered":"2024-07-26T06:15:09.063424+0000","last_clean":"2024-07-26T06:15:09.063424+0000","last_became_active":"2024-07-26T06:15:07.452431+0000","last_became_peered":"2024-07-26T06:15:07.452431+0000","last_unstale":"2024-07-26T06:15:09.063424+0000","last_undegraded":"2024-07-26T06:15:09.063424+0000","last_fullsized":"2024-07-26T06:15:09.063424+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:14:36.028736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442460+0000","last_change":"2024-07-26T06:15:07.438387+0000","last_active":"2024-07-26T06:15:08.442460+0000","last_peered":"2024-07-26T06:15:08.442460+0000","last_clean":"2024-07-26T06:15:08.442460+0000","last_became_active":"2024-07-26T06:15:07.437984+0000","last_became_peered":"2024-07-26T06:15:07.437984+0000","last_unstale":"2024-07-26T06:15:08.442460+0000","last_undegraded":"2024-07-26T06:15:08.442460+0000","last_fullsized":"2024-07-26T06:15:08.442460+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:42:51.944062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432952+0000","last_change":"2024-07-26T06:15:07.435958+0000","last_active":"2024-07-26T06:15:08.432952+0000","last_peered":"2024-07-26T06:15:08.432952+0000","last_clean":"2024-07-26T06:15:08.432952+0000","last_became_active":"2024-07-26T06:15:07.435636+0000","last_became_peered":"2024-07-26T06:15:07.435636+0000","last_unstale":"2024-07-26T06:15:08.432952+0000","last_undegraded":"2024-07-26T06:15:08.432952+0000","last_fullsized":"2024-07-26T06:15:08.432952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:09:06.481920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427528+0000","last_change":"2024-07-26T06:15:07.429783+0000","last_active":"2024-07-26T06:15:08.427528+0000","last_peered":"2024-07-26T06:15:08.427528+0000","last_clean":"2024-07-26T06:15:08.427528+0000","last_became_active":"2024-07-26T06:15:07.429434+0000","last_became_peered":"2024-07-26T06:15:07.429434+0000","last_unstale":"2024-07-26T06:15:08.427528+0000","last_undegraded":"2024-07-26T06:15:08.427528+0000","last_fullsized":"2024-07-26T06:15:08.427528+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:55.220456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162612+0000","last_change":"2024-07-26T06:15:07.428785+0000","last_active":"2024-07-26T06:15:10.162612+0000","last_peered":"2024-07-26T06:15:10.162612+0000","last_clean":"2024-07-26T06:15:10.162612+0000","last_became_active":"2024-07-26T06:15:07.428493+0000","last_became_peered":"2024-07-26T06:15:07.428493+0000","last_unstale":"2024-07-26T06:15:10.162612+0000","last_undegraded":"2024-07-26T06:15:10.162612+0000","last_fullsized":"2024-07-26T06:15:10.162612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:00.268991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432740+0000","last_change":"2024-07-26T06:15:06.428719+0000","last_active":"2024-07-26T06:15:08.432740+0000","last_peered":"2024-07-26T06:15:08.432740+0000","last_clean":"2024-07-26T06:15:08.432740+0000","last_became_active":"2024-07-26T06:15:06.428416+0000","last_became_peered":"2024-07-26T06:15:06.428416+0000","last_unstale":"2024-07-26T06:15:08.432740+0000","last_undegraded":"2024-07-26T06:15:08.432740+0000","last_fullsized":"2024-07-26T06:15:08.432740+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:25.572744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063722+0000","last_change":"2024-07-26T06:15:07.447042+0000","last_active":"2024-07-26T06:15:09.063722+0000","last_peered":"2024-07-26T06:15:09.063722+0000","last_clean":"2024-07-26T06:15:09.063722+0000","last_became_active":"2024-07-26T06:15:07.446737+0000","last_became_peered":"2024-07-26T06:15:07.446737+0000","last_unstale":"2024-07-26T06:15:09.063722+0000","last_undegraded":"2024-07-26T06:15:09.063722+0000","last_fullsized":"2024-07-26T06:15:09.063722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.177718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442883+0000","last_change":"2024-07-26T06:15:06.425064+0000","last_active":"2024-07-26T06:15:08.442883+0000","last_peered":"2024-07-26T06:15:08.442883+0000","last_clean":"2024-07-26T06:15:08.442883+0000","last_became_active":"2024-07-26T06:15:06.424560+0000","last_became_peered":"2024-07-26T06:15:06.424560+0000","last_unstale":"2024-07-26T06:15:08.442883+0000","last_undegraded":"2024-07-26T06:15:08.442883+0000","last_fullsized":"2024-07-26T06:15:08.442883+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:32:52.880734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165852+0000","last_change":"2024-07-26T06:15:07.438595+0000","last_active":"2024-07-26T06:15:10.165852+0000","last_peered":"2024-07-26T06:15:10.165852+0000","last_clean":"2024-07-26T06:15:10.165852+0000","last_became_active":"2024-07-26T06:15:07.438207+0000","last_became_peered":"2024-07-26T06:15:07.438207+0000","last_unstale":"2024-07-26T06:15:10.165852+0000","last_undegraded":"2024-07-26T06:15:10.165852+0000","last_fullsized":"2024-07-26T06:15:10.165852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:30:23.809092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162857+0000","last_change":"2024-07-26T06:15:06.428424+0000","last_active":"2024-07-26T06:15:10.162857+0000","last_peered":"2024-07-26T06:15:10.162857+0000","last_clean":"2024-07-26T06:15:10.162857+0000","last_became_active":"2024-07-26T06:15:06.428100+0000","last_became_peered":"2024-07-26T06:15:06.428100+0000","last_unstale":"2024-07-26T06:15:10.162857+0000","last_undegraded":"2024-07-26T06:15:10.162857+0000","last_fullsized":"2024-07-26T06:15:10.162857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:32:52.880734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.163002+0000","last_change":"2024-07-26T06:15:07.456057+0000","last_active":"2024-07-26T06:15:10.163002+0000","last_peered":"2024-07-26T06:15:10.163002+0000","last_clean":"2024-07-26T06:15:10.163002+0000","last_became_active":"2024-07-26T06:15:07.455730+0000","last_became_peered":"2024-07-26T06:15:07.455730+0000","last_unstale":"2024-07-26T06:15:10.163002+0000","last_undegraded":"2024-07-26T06:15:10.163002+0000","last_fullsized":"2024-07-26T06:15:10.163002+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:18:56.750715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428991+0000","last_change":"2024-07-26T06:15:06.434477+0000","last_active":"2024-07-26T06:15:08.428991+0000","last_peered":"2024-07-26T06:15:08.428991+0000","last_clean":"2024-07-26T06:15:08.428991+0000","last_became_active":"2024-07-26T06:15:06.433902+0000","last_became_peered":"2024-07-26T06:15:06.433902+0000","last_unstale":"2024-07-26T06:15:08.428991+0000","last_undegraded":"2024-07-26T06:15:08.428991+0000","last_fullsized":"2024-07-26T06:15:08.428991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:24:57.104664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442227+0000","last_change":"2024-07-26T06:15:07.446994+0000","last_active":"2024-07-26T06:15:08.442227+0000","last_peered":"2024-07-26T06:15:08.442227+0000","last_clean":"2024-07-26T06:15:08.442227+0000","last_became_active":"2024-07-26T06:15:07.446724+0000","last_became_peered":"2024-07-26T06:15:07.446724+0000","last_unstale":"2024-07-26T06:15:08.442227+0000","last_undegraded":"2024-07-26T06:15:08.442227+0000","last_fullsized":"2024-07-26T06:15:08.442227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:00.713979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431870+0000","last_change":"2024-07-26T06:15:06.430887+0000","last_active":"2024-07-26T06:15:08.431870+0000","last_peered":"2024-07-26T06:15:08.431870+0000","last_clean":"2024-07-26T06:15:08.431870+0000","last_became_active":"2024-07-26T06:15:06.429962+0000","last_became_peered":"2024-07-26T06:15:06.429962+0000","last_unstale":"2024-07-26T06:15:08.431870+0000","last_undegraded":"2024-07-26T06:15:08.431870+0000","last_fullsized":"2024-07-26T06:15:08.431870+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:54:06.599271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428762+0000","last_change":"2024-07-26T06:15:07.432241+0000","last_active":"2024-07-26T06:15:08.428762+0000","last_peered":"2024-07-26T06:15:08.428762+0000","last_clean":"2024-07-26T06:15:08.428762+0000","last_became_active":"2024-07-26T06:15:07.431906+0000","last_became_peered":"2024-07-26T06:15:07.431906+0000","last_unstale":"2024-07-26T06:15:08.428762+0000","last_undegraded":"2024-07-26T06:15:08.428762+0000","last_fullsized":"2024-07-26T06:15:08.428762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:09.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442398+0000","last_change":"2024-07-26T06:15:06.431877+0000","last_active":"2024-07-26T06:15:08.442398+0000","last_peered":"2024-07-26T06:15:08.442398+0000","last_clean":"2024-07-26T06:15:08.442398+0000","last_became_active":"2024-07-26T06:15:06.431309+0000","last_became_peered":"2024-07-26T06:15:06.431309+0000","last_unstale":"2024-07-26T06:15:08.442398+0000","last_undegraded":"2024-07-26T06:15:08.442398+0000","last_fullsized":"2024-07-26T06:15:08.442398+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:03:46.856382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427311+0000","last_change":"2024-07-26T06:15:07.446522+0000","last_active":"2024-07-26T06:15:08.427311+0000","last_peered":"2024-07-26T06:15:08.427311+0000","last_clean":"2024-07-26T06:15:08.427311+0000","last_became_active":"2024-07-26T06:15:07.446199+0000","last_became_peered":"2024-07-26T06:15:07.446199+0000","last_unstale":"2024-07-26T06:15:08.427311+0000","last_undegraded":"2024-07-26T06:15:08.427311+0000","last_fullsized":"2024-07-26T06:15:08.427311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:41.575875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165173+0000","last_change":"2024-07-26T06:15:06.428513+0000","last_active":"2024-07-26T06:15:10.165173+0000","last_peered":"2024-07-26T06:15:10.165173+0000","last_clean":"2024-07-26T06:15:10.165173+0000","last_became_active":"2024-07-26T06:15:06.427510+0000","last_became_peered":"2024-07-26T06:15:06.427510+0000","last_unstale":"2024-07-26T06:15:10.165173+0000","last_undegraded":"2024-07-26T06:15:10.165173+0000","last_fullsized":"2024-07-26T06:15:10.165173+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:34.116708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063181+0000","last_change":"2024-07-26T06:15:07.453292+0000","last_active":"2024-07-26T06:15:09.063181+0000","last_peered":"2024-07-26T06:15:09.063181+0000","last_clean":"2024-07-26T06:15:09.063181+0000","last_became_active":"2024-07-26T06:15:07.451899+0000","last_became_peered":"2024-07-26T06:15:07.451899+0000","last_unstale":"2024-07-26T06:15:09.063181+0000","last_undegraded":"2024-07-26T06:15:09.063181+0000","last_fullsized":"2024-07-26T06:15:09.063181+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:05.145115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427367+0000","last_change":"2024-07-26T06:15:06.416806+0000","last_active":"2024-07-26T06:15:08.427367+0000","last_peered":"2024-07-26T06:15:08.427367+0000","last_clean":"2024-07-26T06:15:08.427367+0000","last_became_active":"2024-07-26T06:15:06.415836+0000","last_became_peered":"2024-07-26T06:15:06.415836+0000","last_unstale":"2024-07-26T06:15:08.427367+0000","last_undegraded":"2024-07-26T06:15:08.427367+0000","last_fullsized":"2024-07-26T06:15:08.427367+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:44:54.680418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442955+0000","last_change":"2024-07-26T06:15:07.427313+0000","last_active":"2024-07-26T06:15:08.442955+0000","last_peered":"2024-07-26T06:15:08.442955+0000","last_clean":"2024-07-26T06:15:08.442955+0000","last_became_active":"2024-07-26T06:15:07.427007+0000","last_became_peered":"2024-07-26T06:15:07.427007+0000","last_unstale":"2024-07-26T06:15:08.442955+0000","last_undegraded":"2024-07-26T06:15:08.442955+0000","last_fullsized":"2024-07-26T06:15:08.442955+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:26.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442902+0000","last_change":"2024-07-26T06:15:06.418583+0000","last_active":"2024-07-26T06:15:08.442902+0000","last_peered":"2024-07-26T06:15:08.442902+0000","last_clean":"2024-07-26T06:15:08.442902+0000","last_became_active":"2024-07-26T06:15:06.417671+0000","last_became_peered":"2024-07-26T06:15:06.417671+0000","last_unstale":"2024-07-26T06:15:08.442902+0000","last_undegraded":"2024-07-26T06:15:08.442902+0000","last_fullsized":"2024-07-26T06:15:08.442902+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:58:07.482287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429547+0000","last_change":"2024-07-26T06:15:07.449855+0000","last_active":"2024-07-26T06:15:08.429547+0000","last_peered":"2024-07-26T06:15:08.429547+0000","last_clean":"2024-07-26T06:15:08.429547+0000","last_became_active":"2024-07-26T06:15:07.449557+0000","last_became_peered":"2024-07-26T06:15:07.449557+0000","last_unstale":"2024-07-26T06:15:08.429547+0000","last_undegraded":"2024-07-26T06:15:08.429547+0000","last_fullsized":"2024-07-26T06:15:08.429547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:01:48.971591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430340+0000","last_change":"2024-07-26T06:15:06.427881+0000","last_active":"2024-07-26T06:15:08.430340+0000","last_peered":"2024-07-26T06:15:08.430340+0000","last_clean":"2024-07-26T06:15:08.430340+0000","last_became_active":"2024-07-26T06:15:06.427244+0000","last_became_peered":"2024-07-26T06:15:06.427244+0000","last_unstale":"2024-07-26T06:15:08.430340+0000","last_undegraded":"2024-07-26T06:15:08.430340+0000","last_fullsized":"2024-07-26T06:15:08.430340+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:31.606939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432752+0000","last_change":"2024-07-26T06:15:07.433401+0000","last_active":"2024-07-26T06:15:08.432752+0000","last_peered":"2024-07-26T06:15:08.432752+0000","last_clean":"2024-07-26T06:15:08.432752+0000","last_became_active":"2024-07-26T06:15:07.432643+0000","last_became_peered":"2024-07-26T06:15:07.432643+0000","last_unstale":"2024-07-26T06:15:08.432752+0000","last_undegraded":"2024-07-26T06:15:08.432752+0000","last_fullsized":"2024-07-26T06:15:08.432752+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:09.016877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429480+0000","last_change":"2024-07-26T06:15:06.440211+0000","last_active":"2024-07-26T06:15:08.429480+0000","last_peered":"2024-07-26T06:15:08.429480+0000","last_clean":"2024-07-26T06:15:08.429480+0000","last_became_active":"2024-07-26T06:15:06.439822+0000","last_became_peered":"2024-07-26T06:15:06.439822+0000","last_unstale":"2024-07-26T06:15:08.429480+0000","last_undegraded":"2024-07-26T06:15:08.429480+0000","last_fullsized":"2024-07-26T06:15:08.429480+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:08.567851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429209+0000","last_change":"2024-07-26T06:15:07.441660+0000","last_active":"2024-07-26T06:15:08.429209+0000","last_peered":"2024-07-26T06:15:08.429209+0000","last_clean":"2024-07-26T06:15:08.429209+0000","last_became_active":"2024-07-26T06:15:07.441406+0000","last_became_peered":"2024-07-26T06:15:07.441406+0000","last_unstale":"2024-07-26T06:15:08.429209+0000","last_undegraded":"2024-07-26T06:15:08.429209+0000","last_fullsized":"2024-07-26T06:15:08.429209+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:03:35.230183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431920+0000","last_change":"2024-07-26T06:15:06.421856+0000","last_active":"2024-07-26T06:15:08.431920+0000","last_peered":"2024-07-26T06:15:08.431920+0000","last_clean":"2024-07-26T06:15:08.431920+0000","last_became_active":"2024-07-26T06:15:06.421246+0000","last_became_peered":"2024-07-26T06:15:06.421246+0000","last_unstale":"2024-07-26T06:15:08.431920+0000","last_undegraded":"2024-07-26T06:15:08.431920+0000","last_fullsized":"2024-07-26T06:15:08.431920+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:21.906289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063131+0000","last_change":"2024-07-26T06:15:07.453126+0000","last_active":"2024-07-26T06:15:09.063131+0000","last_peered":"2024-07-26T06:15:09.063131+0000","last_clean":"2024-07-26T06:15:09.063131+0000","last_became_active":"2024-07-26T06:15:07.451881+0000","last_became_peered":"2024-07-26T06:15:07.451881+0000","last_unstale":"2024-07-26T06:15:09.063131+0000","last_undegraded":"2024-07-26T06:15:09.063131+0000","last_fullsized":"2024-07-26T06:15:09.063131+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:03.327048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432445+0000","last_change":"2024-07-26T06:15:06.425695+0000","last_active":"2024-07-26T06:15:08.432445+0000","last_peered":"2024-07-26T06:15:08.432445+0000","last_clean":"2024-07-26T06:15:08.432445+0000","last_became_active":"2024-07-26T06:15:06.425385+0000","last_became_peered":"2024-07-26T06:15:06.425385+0000","last_unstale":"2024-07-26T06:15:08.432445+0000","last_undegraded":"2024-07-26T06:15:08.432445+0000","last_fullsized":"2024-07-26T06:15:08.432445+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:50:25.392213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429021+0000","last_change":"2024-07-26T06:15:07.440623+0000","last_active":"2024-07-26T06:15:08.429021+0000","last_peered":"2024-07-26T06:15:08.429021+0000","last_clean":"2024-07-26T06:15:08.429021+0000","last_became_active":"2024-07-26T06:15:07.440297+0000","last_became_peered":"2024-07-26T06:15:07.440297+0000","last_unstale":"2024-07-26T06:15:08.429021+0000","last_undegraded":"2024-07-26T06:15:08.429021+0000","last_fullsized":"2024-07-26T06:15:08.429021+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:51.343615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428657+0000","last_change":"2024-07-26T06:15:06.430182+0000","last_active":"2024-07-26T06:15:08.428657+0000","last_peered":"2024-07-26T06:15:08.428657+0000","last_clean":"2024-07-26T06:15:08.428657+0000","last_became_active":"2024-07-26T06:15:06.429864+0000","last_became_peered":"2024-07-26T06:15:06.429864+0000","last_unstale":"2024-07-26T06:15:08.428657+0000","last_undegraded":"2024-07-26T06:15:08.428657+0000","last_fullsized":"2024-07-26T06:15:08.428657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:34.116708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431100+0000","last_change":"2024-07-26T06:15:07.432954+0000","last_active":"2024-07-26T06:15:08.431100+0000","last_peered":"2024-07-26T06:15:08.431100+0000","last_clean":"2024-07-26T06:15:08.431100+0000","last_became_active":"2024-07-26T06:15:07.432654+0000","last_became_peered":"2024-07-26T06:15:07.432654+0000","last_unstale":"2024-07-26T06:15:08.431100+0000","last_undegraded":"2024-07-26T06:15:08.431100+0000","last_fullsized":"2024-07-26T06:15:08.431100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:46:09.745514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432280+0000","last_change":"2024-07-26T06:15:06.425752+0000","last_active":"2024-07-26T06:15:08.432280+0000","last_peered":"2024-07-26T06:15:08.432280+0000","last_clean":"2024-07-26T06:15:08.432280+0000","last_became_active":"2024-07-26T06:15:06.425005+0000","last_became_peered":"2024-07-26T06:15:06.425005+0000","last_unstale":"2024-07-26T06:15:08.432280+0000","last_undegraded":"2024-07-26T06:15:08.432280+0000","last_fullsized":"2024-07-26T06:15:08.432280+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:36:12.256587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428351+0000","last_change":"2024-07-26T06:15:07.457696+0000","last_active":"2024-07-26T06:15:08.428351+0000","last_peered":"2024-07-26T06:15:08.428351+0000","last_clean":"2024-07-26T06:15:08.428351+0000","last_became_active":"2024-07-26T06:15:07.454380+0000","last_became_peered":"2024-07-26T06:15:07.454380+0000","last_unstale":"2024-07-26T06:15:08.428351+0000","last_undegraded":"2024-07-26T06:15:08.428351+0000","last_fullsized":"2024-07-26T06:15:08.428351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:21:23.009970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427661+0000","last_change":"2024-07-26T06:15:06.427250+0000","last_active":"2024-07-26T06:15:08.427661+0000","last_peered":"2024-07-26T06:15:08.427661+0000","last_clean":"2024-07-26T06:15:08.427661+0000","last_became_active":"2024-07-26T06:15:06.426832+0000","last_became_peered":"2024-07-26T06:15:06.426832+0000","last_unstale":"2024-07-26T06:15:08.427661+0000","last_undegraded":"2024-07-26T06:15:08.427661+0000","last_fullsized":"2024-07-26T06:15:08.427661+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:54.214574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430516+0000","last_change":"2024-07-26T06:15:07.442796+0000","last_active":"2024-07-26T06:15:08.430516+0000","last_peered":"2024-07-26T06:15:08.430516+0000","last_clean":"2024-07-26T06:15:08.430516+0000","last_became_active":"2024-07-26T06:15:07.440503+0000","last_became_peered":"2024-07-26T06:15:07.440503+0000","last_unstale":"2024-07-26T06:15:08.430516+0000","last_undegraded":"2024-07-26T06:15:08.430516+0000","last_fullsized":"2024-07-26T06:15:08.430516+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:17.798570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.433106+0000","last_change":"2024-07-26T06:15:06.421238+0000","last_active":"2024-07-26T06:15:08.433106+0000","last_peered":"2024-07-26T06:15:08.433106+0000","last_clean":"2024-07-26T06:15:08.433106+0000","last_became_active":"2024-07-26T06:15:06.420892+0000","last_became_peered":"2024-07-26T06:15:06.420892+0000","last_unstale":"2024-07-26T06:15:08.433106+0000","last_undegraded":"2024-07-26T06:15:08.433106+0000","last_fullsized":"2024-07-26T06:15:08.433106+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:39.822740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430381+0000","last_change":"2024-07-26T06:15:07.421052+0000","last_active":"2024-07-26T06:15:08.430381+0000","last_peered":"2024-07-26T06:15:08.430381+0000","last_clean":"2024-07-26T06:15:08.430381+0000","last_became_active":"2024-07-26T06:15:07.420689+0000","last_became_peered":"2024-07-26T06:15:07.420689+0000","last_unstale":"2024-07-26T06:15:08.430381+0000","last_undegraded":"2024-07-26T06:15:08.430381+0000","last_fullsized":"2024-07-26T06:15:08.430381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:44.629861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.443041+0000","last_change":"2024-07-26T06:15:06.413462+0000","last_active":"2024-07-26T06:15:08.443041+0000","last_peered":"2024-07-26T06:15:08.443041+0000","last_clean":"2024-07-26T06:15:08.443041+0000","last_became_active":"2024-07-26T06:15:06.413050+0000","last_became_peered":"2024-07-26T06:15:06.413050+0000","last_unstale":"2024-07-26T06:15:08.443041+0000","last_undegraded":"2024-07-26T06:15:08.443041+0000","last_fullsized":"2024-07-26T06:15:08.443041+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:55:26.398849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063616+0000","last_change":"2024-07-26T06:15:07.452021+0000","last_active":"2024-07-26T06:15:09.063616+0000","last_peered":"2024-07-26T06:15:09.063616+0000","last_clean":"2024-07-26T06:15:09.063616+0000","last_became_active":"2024-07-26T06:15:07.451697+0000","last_became_peered":"2024-07-26T06:15:07.451697+0000","last_unstale":"2024-07-26T06:15:09.063616+0000","last_undegraded":"2024-07-26T06:15:09.063616+0000","last_fullsized":"2024-07-26T06:15:09.063616+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:50.773867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165583+0000","last_change":"2024-07-26T06:15:06.434157+0000","last_active":"2024-07-26T06:15:10.165583+0000","last_peered":"2024-07-26T06:15:10.165583+0000","last_clean":"2024-07-26T06:15:10.165583+0000","last_became_active":"2024-07-26T06:15:06.433764+0000","last_became_peered":"2024-07-26T06:15:06.433764+0000","last_unstale":"2024-07-26T06:15:10.165583+0000","last_undegraded":"2024-07-26T06:15:10.165583+0000","last_fullsized":"2024-07-26T06:15:10.165583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:55.743833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063214+0000","last_change":"2024-07-26T06:15:07.453298+0000","last_active":"2024-07-26T06:15:09.063214+0000","last_peered":"2024-07-26T06:15:09.063214+0000","last_clean":"2024-07-26T06:15:09.063214+0000","last_became_active":"2024-07-26T06:15:07.452296+0000","last_became_peered":"2024-07-26T06:15:07.452296+0000","last_unstale":"2024-07-26T06:15:09.063214+0000","last_undegraded":"2024-07-26T06:15:09.063214+0000","last_fullsized":"2024-07-26T06:15:09.063214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:38.857932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432483+0000","last_change":"2024-07-26T06:15:06.431291+0000","last_active":"2024-07-26T06:15:08.432483+0000","last_peered":"2024-07-26T06:15:08.432483+0000","last_clean":"2024-07-26T06:15:08.432483+0000","last_became_active":"2024-07-26T06:15:06.430987+0000","last_became_peered":"2024-07-26T06:15:06.430987+0000","last_unstale":"2024-07-26T06:15:08.432483+0000","last_undegraded":"2024-07-26T06:15:08.432483+0000","last_fullsized":"2024-07-26T06:15:08.432483+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:27.051477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431531+0000","last_change":"2024-07-26T06:15:07.428559+0000","last_active":"2024-07-26T06:15:08.431531+0000","last_peered":"2024-07-26T06:15:08.431531+0000","last_clean":"2024-07-26T06:15:08.431531+0000","last_became_active":"2024-07-26T06:15:07.427951+0000","last_became_peered":"2024-07-26T06:15:07.427951+0000","last_unstale":"2024-07-26T06:15:08.431531+0000","last_undegraded":"2024-07-26T06:15:08.431531+0000","last_fullsized":"2024-07-26T06:15:08.431531+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:41.845575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.433070+0000","last_change":"2024-07-26T06:15:06.431511+0000","last_active":"2024-07-26T06:15:08.433070+0000","last_peered":"2024-07-26T06:15:08.433070+0000","last_clean":"2024-07-26T06:15:08.433070+0000","last_became_active":"2024-07-26T06:15:06.430043+0000","last_became_peered":"2024-07-26T06:15:06.430043+0000","last_unstale":"2024-07-26T06:15:08.433070+0000","last_undegraded":"2024-07-26T06:15:08.433070+0000","last_fullsized":"2024-07-26T06:15:08.433070+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:21:25.074600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162693+0000","last_change":"2024-07-26T06:15:07.431238+0000","last_active":"2024-07-26T06:15:10.162693+0000","last_peered":"2024-07-26T06:15:10.162693+0000","last_clean":"2024-07-26T06:15:10.162693+0000","last_became_active":"2024-07-26T06:15:07.430655+0000","last_became_peered":"2024-07-26T06:15:07.430655+0000","last_unstale":"2024-07-26T06:15:10.162693+0000","last_undegraded":"2024-07-26T06:15:10.162693+0000","last_fullsized":"2024-07-26T06:15:10.162693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:39:41.396598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.164887+0000","last_change":"2024-07-26T06:15:06.432150+0000","last_active":"2024-07-26T06:15:10.164887+0000","last_peered":"2024-07-26T06:15:10.164887+0000","last_clean":"2024-07-26T06:15:10.164887+0000","last_became_active":"2024-07-26T06:15:06.431043+0000","last_became_peered":"2024-07-26T06:15:06.431043+0000","last_unstale":"2024-07-26T06:15:10.164887+0000","last_undegraded":"2024-07-26T06:15:10.164887+0000","last_fullsized":"2024-07-26T06:15:10.164887+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:40.569992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427922+0000","last_change":"2024-07-26T06:15:07.445135+0000","last_active":"2024-07-26T06:15:08.427922+0000","last_peered":"2024-07-26T06:15:08.427922+0000","last_clean":"2024-07-26T06:15:08.427922+0000","last_became_active":"2024-07-26T06:15:07.444835+0000","last_became_peered":"2024-07-26T06:15:07.444835+0000","last_unstale":"2024-07-26T06:15:08.427922+0000","last_undegraded":"2024-07-26T06:15:08.427922+0000","last_fullsized":"2024-07-26T06:15:08.427922+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:27.950292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427439+0000","last_change":"2024-07-26T06:15:06.428563+0000","last_active":"2024-07-26T06:15:08.427439+0000","last_peered":"2024-07-26T06:15:08.427439+0000","last_clean":"2024-07-26T06:15:08.427439+0000","last_became_active":"2024-07-26T06:15:06.427905+0000","last_became_peered":"2024-07-26T06:15:06.427905+0000","last_unstale":"2024-07-26T06:15:08.427439+0000","last_undegraded":"2024-07-26T06:15:08.427439+0000","last_fullsized":"2024-07-26T06:15:08.427439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:07:43.751746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427140+0000","last_change":"2024-07-26T06:15:07.427333+0000","last_active":"2024-07-26T06:15:08.427140+0000","last_peered":"2024-07-26T06:15:08.427140+0000","last_clean":"2024-07-26T06:15:08.427140+0000","last_became_active":"2024-07-26T06:15:07.426998+0000","last_became_peered":"2024-07-26T06:15:07.426998+0000","last_unstale":"2024-07-26T06:15:08.427140+0000","last_undegraded":"2024-07-26T06:15:08.427140+0000","last_fullsized":"2024-07-26T06:15:08.427140+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:53:26.197289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427912+0000","last_change":"2024-07-26T06:15:06.429477+0000","last_active":"2024-07-26T06:15:08.427912+0000","last_peered":"2024-07-26T06:15:08.427912+0000","last_clean":"2024-07-26T06:15:08.427912+0000","last_became_active":"2024-07-26T06:15:06.429001+0000","last_became_peered":"2024-07-26T06:15:06.429001+0000","last_unstale":"2024-07-26T06:15:08.427912+0000","last_undegraded":"2024-07-26T06:15:08.427912+0000","last_fullsized":"2024-07-26T06:15:08.427912+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:50.337732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063856+0000","last_change":"2024-07-26T06:15:07.447128+0000","last_active":"2024-07-26T06:15:09.063856+0000","last_peered":"2024-07-26T06:15:09.063856+0000","last_clean":"2024-07-26T06:15:09.063856+0000","last_became_active":"2024-07-26T06:15:07.446757+0000","last_became_peered":"2024-07-26T06:15:07.446757+0000","last_unstale":"2024-07-26T06:15:09.063856+0000","last_undegraded":"2024-07-26T06:15:09.063856+0000","last_fullsized":"2024-07-26T06:15:09.063856+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:16:49.612701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432647+0000","last_change":"2024-07-26T06:15:06.423800+0000","last_active":"2024-07-26T06:15:08.432647+0000","last_peered":"2024-07-26T06:15:08.432647+0000","last_clean":"2024-07-26T06:15:08.432647+0000","last_became_active":"2024-07-26T06:15:06.423480+0000","last_became_peered":"2024-07-26T06:15:06.423480+0000","last_unstale":"2024-07-26T06:15:08.432647+0000","last_undegraded":"2024-07-26T06:15:08.432647+0000","last_fullsized":"2024-07-26T06:15:08.432647+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:24.551356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428971+0000","last_change":"2024-07-26T06:15:07.415883+0000","last_active":"2024-07-26T06:15:08.428971+0000","last_peered":"2024-07-26T06:15:08.428971+0000","last_clean":"2024-07-26T06:15:08.428971+0000","last_became_active":"2024-07-26T06:15:07.415464+0000","last_became_peered":"2024-07-26T06:15:07.415464+0000","last_unstale":"2024-07-26T06:15:08.428971+0000","last_undegraded":"2024-07-26T06:15:08.428971+0000","last_fullsized":"2024-07-26T06:15:08.428971+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:58.101987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429649+0000","last_change":"2024-07-26T06:15:06.440175+0000","last_active":"2024-07-26T06:15:08.429649+0000","last_peered":"2024-07-26T06:15:08.429649+0000","last_clean":"2024-07-26T06:15:08.429649+0000","last_became_active":"2024-07-26T06:15:06.439829+0000","last_became_peered":"2024-07-26T06:15:06.439829+0000","last_unstale":"2024-07-26T06:15:08.429649+0000","last_undegraded":"2024-07-26T06:15:08.429649+0000","last_fullsized":"2024-07-26T06:15:08.429649+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:57.096105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429434+0000","last_change":"2024-07-26T06:15:07.437465+0000","last_active":"2024-07-26T06:15:08.429434+0000","last_peered":"2024-07-26T06:15:08.429434+0000","last_clean":"2024-07-26T06:15:08.429434+0000","last_became_active":"2024-07-26T06:15:07.437100+0000","last_became_peered":"2024-07-26T06:15:07.437100+0000","last_unstale":"2024-07-26T06:15:08.429434+0000","last_undegraded":"2024-07-26T06:15:08.429434+0000","last_fullsized":"2024-07-26T06:15:08.429434+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:28:36.803795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162482+0000","last_change":"2024-07-26T06:15:06.428572+0000","last_active":"2024-07-26T06:15:10.162482+0000","last_peered":"2024-07-26T06:15:10.162482+0000","last_clean":"2024-07-26T06:15:10.162482+0000","last_became_active":"2024-07-26T06:15:06.428279+0000","last_became_peered":"2024-07-26T06:15:06.428279+0000","last_unstale":"2024-07-26T06:15:10.162482+0000","last_undegraded":"2024-07-26T06:15:10.162482+0000","last_fullsized":"2024-07-26T06:15:10.162482+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:50:49.781379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431623+0000","last_change":"2024-07-26T06:15:07.428752+0000","last_active":"2024-07-26T06:15:08.431623+0000","last_peered":"2024-07-26T06:15:08.431623+0000","last_clean":"2024-07-26T06:15:08.431623+0000","last_became_active":"2024-07-26T06:15:07.428327+0000","last_became_peered":"2024-07-26T06:15:07.428327+0000","last_unstale":"2024-07-26T06:15:08.431623+0000","last_undegraded":"2024-07-26T06:15:08.431623+0000","last_fullsized":"2024-07-26T06:15:08.431623+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:07.323166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428956+0000","last_change":"2024-07-26T06:15:06.434053+0000","last_active":"2024-07-26T06:15:08.428956+0000","last_peered":"2024-07-26T06:15:08.428956+0000","last_clean":"2024-07-26T06:15:08.428956+0000","last_became_active":"2024-07-26T06:15:06.433743+0000","last_became_peered":"2024-07-26T06:15:06.433743+0000","last_unstale":"2024-07-26T06:15:08.428956+0000","last_undegraded":"2024-07-26T06:15:08.428956+0000","last_fullsized":"2024-07-26T06:15:08.428956+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:55.743833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432303+0000","last_change":"2024-07-26T06:15:07.434188+0000","last_active":"2024-07-26T06:15:08.432303+0000","last_peered":"2024-07-26T06:15:08.432303+0000","last_clean":"2024-07-26T06:15:08.432303+0000","last_became_active":"2024-07-26T06:15:07.433895+0000","last_became_peered":"2024-07-26T06:15:07.433895+0000","last_unstale":"2024-07-26T06:15:08.432303+0000","last_undegraded":"2024-07-26T06:15:08.432303+0000","last_fullsized":"2024-07-26T06:15:08.432303+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:08.498466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442622+0000","last_change":"2024-07-26T06:15:06.419916+0000","last_active":"2024-07-26T06:15:08.442622+0000","last_peered":"2024-07-26T06:15:08.442622+0000","last_clean":"2024-07-26T06:15:08.442622+0000","last_became_active":"2024-07-26T06:15:06.419614+0000","last_became_peered":"2024-07-26T06:15:06.419614+0000","last_unstale":"2024-07-26T06:15:08.442622+0000","last_undegraded":"2024-07-26T06:15:08.442622+0000","last_fullsized":"2024-07-26T06:15:08.442622+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:50:49.781379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165051+0000","last_change":"2024-07-26T06:15:07.441831+0000","last_active":"2024-07-26T06:15:10.165051+0000","last_peered":"2024-07-26T06:15:10.165051+0000","last_clean":"2024-07-26T06:15:10.165051+0000","last_became_active":"2024-07-26T06:15:07.440883+0000","last_became_peered":"2024-07-26T06:15:07.440883+0000","last_unstale":"2024-07-26T06:15:10.165051+0000","last_undegraded":"2024-07-26T06:15:10.165051+0000","last_fullsized":"2024-07-26T06:15:10.165051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:07:02.236299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162501+0000","last_change":"2024-07-26T06:15:06.424010+0000","last_active":"2024-07-26T06:15:10.162501+0000","last_peered":"2024-07-26T06:15:10.162501+0000","last_clean":"2024-07-26T06:15:10.162501+0000","last_became_active":"2024-07-26T06:15:06.423692+0000","last_became_peered":"2024-07-26T06:15:06.423692+0000","last_unstale":"2024-07-26T06:15:10.162501+0000","last_undegraded":"2024-07-26T06:15:10.162501+0000","last_fullsized":"2024-07-26T06:15:10.162501+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:03:46.856382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442216+0000","last_change":"2024-07-26T06:15:07.416039+0000","last_active":"2024-07-26T06:15:08.442216+0000","last_peered":"2024-07-26T06:15:08.442216+0000","last_clean":"2024-07-26T06:15:08.442216+0000","last_became_active":"2024-07-26T06:15:07.414833+0000","last_became_peered":"2024-07-26T06:15:07.414833+0000","last_unstale":"2024-07-26T06:15:08.442216+0000","last_undegraded":"2024-07-26T06:15:08.442216+0000","last_fullsized":"2024-07-26T06:15:08.442216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:37:22.010597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428007+0000","last_change":"2024-07-26T06:15:06.425341+0000","last_active":"2024-07-26T06:15:08.428007+0000","last_peered":"2024-07-26T06:15:08.428007+0000","last_clean":"2024-07-26T06:15:08.428007+0000","last_became_active":"2024-07-26T06:15:06.424843+0000","last_became_peered":"2024-07-26T06:15:06.424843+0000","last_unstale":"2024-07-26T06:15:08.428007+0000","last_undegraded":"2024-07-26T06:15:08.428007+0000","last_fullsized":"2024-07-26T06:15:08.428007+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:26.944410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430123+0000","last_change":"2024-07-26T06:15:07.441572+0000","last_active":"2024-07-26T06:15:08.430123+0000","last_peered":"2024-07-26T06:15:08.430123+0000","last_clean":"2024-07-26T06:15:08.430123+0000","last_became_active":"2024-07-26T06:15:07.440083+0000","last_became_peered":"2024-07-26T06:15:07.440083+0000","last_unstale":"2024-07-26T06:15:08.430123+0000","last_undegraded":"2024-07-26T06:15:08.430123+0000","last_fullsized":"2024-07-26T06:15:08.430123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:07:02.236299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.166175+0000","last_change":"2024-07-26T06:15:06.428716+0000","last_active":"2024-07-26T06:15:10.166175+0000","last_peered":"2024-07-26T06:15:10.166175+0000","last_clean":"2024-07-26T06:15:10.166175+0000","last_became_active":"2024-07-26T06:15:06.427613+0000","last_became_peered":"2024-07-26T06:15:06.427613+0000","last_unstale":"2024-07-26T06:15:10.166175+0000","last_undegraded":"2024-07-26T06:15:10.166175+0000","last_fullsized":"2024-07-26T06:15:10.166175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:53:25.191407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":153,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428127+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:15:08.428127+0000","last_peered":"2024-07-26T06:15:08.428127+0000","last_clean":"2024-07-26T06:15:08.428127+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:15:08.428127+0000","last_undegraded":"2024-07-26T06:15:08.428127+0000","last_fullsized":"2024-07-26T06:15:08.428127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":801163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":211377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":210775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":536576,"data_stored":212339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509466,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":801107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":212317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":211343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":210775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:15:13.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:15:14.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 -- 172.21.15.163:0/4118426113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b840cc1c0 msgr2=0x7f6b840cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 --2- 172.21.15.163:0/4118426113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b840cc1c0 0x7f6b840cd0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009830 tx=0x7f6b7002a5b0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 -- 172.21.15.163:0/4118426113 shutdown_connections 2024-07-26T06:15:14.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 --2- 172.21.15.163:0/4118426113 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b840cc1c0 0x7f6b840cd0c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 -- 172.21.15.163:0/4118426113 >> 172.21.15.163:0/4118426113 conn(0x7f6b840b0e30 msgr2=0x7f6b840b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:14.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 -- 172.21.15.163:0/4118426113 shutdown_connections 2024-07-26T06:15:14.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.101+0000 7f6b8b000640 1 -- 172.21.15.163:0/4118426113 wait complete. 2024-07-26T06:15:14.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.103+0000 7f6b8b000640 1 Processor -- start 2024-07-26T06:15:14.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.103+0000 7f6b8b000640 1 -- start start 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b840741f0 0x7f6b8414f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b840cc1c0 0x7f6b8415c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 0x7f6b841f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8b000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840b25c0 con 0x7f6b8415a6b0 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8b000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840b2440 con 0x7f6b840cc1c0 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8b000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6b840b2740 con 0x7f6b840741f0 2024-07-26T06:15:14.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 0x7f6b841f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 0x7f6b841f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55442/0 (socket says 172.21.15.163:55442) 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 -- 172.21.15.163:0/3304102622 learned_addr learned my addr 172.21.15.163:0/3304102622 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b89200640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b840cc1c0 0x7f6b8415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b89c00640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b840741f0 0x7f6b8414f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 -- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b840741f0 msgr2=0x7f6b8414f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b840741f0 0x7f6b8414f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 -- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b840cc1c0 msgr2=0x7f6b8415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b840cc1c0 0x7f6b8415c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b8a600640 1 -- 172.21.15.163:0/3304102622 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b841f46d0 con 0x7f6b8415a6b0 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.104+0000 7f6b89c00640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b840741f0 0x7f6b8414f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.105+0000 7f6b8a600640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 0x7f6b841f3d00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00c7a0 tx=0x7f6b7c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.105+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c003b70 con 0x7f6b8415a6b0 2024-07-26T06:15:14.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.105+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b7c003d10 con 0x7f6b8415a6b0 2024-07-26T06:15:14.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.105+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b841f49b0 con 0x7f6b8415a6b0 2024-07-26T06:15:14.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.105+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00aab0 con 0x7f6b8415a6b0 2024-07-26T06:15:14.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.105+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b841e3100 con 0x7f6b8415a6b0 2024-07-26T06:15:14.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.106+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b840cd0c0 con 0x7f6b8415a6b0 2024-07-26T06:15:14.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.107+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f6b7c00ac50 con 0x7f6b8415a6b0 2024-07-26T06:15:14.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.108+0000 7f6b81600640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6b64076480 0x7f6b64078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.109+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f6b7c097a30 con 0x7f6b8415a6b0 2024-07-26T06:15:14.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.109+0000 7f6b89c00640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6b64076480 0x7f6b64078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.110+0000 7f6b89c00640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6b64076480 0x7f6b64078940 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b84071fa0 tx=0x7f6b78009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:14.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.113+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b7c061070 con 0x7f6b8415a6b0 2024-07-26T06:15:14.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.314+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b840b4aa0 con 0x7f6b64076480 2024-07-26T06:15:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.327+0000 7f6b81600640 1 -- 172.21.15.163:0/3304102622 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f6b840b4aa0 con 0x7f6b64076480 2024-07-26T06:15:14.332 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:14.334 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:15:14.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6b64076480 msgr2=0x7f6b64078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6b64076480 0x7f6b64078940 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b84071fa0 tx=0x7f6b78009040 comp rx=0 tx=0).stop 2024-07-26T06:15:14.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 msgr2=0x7f6b841f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 0x7f6b841f3d00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00c7a0 tx=0x7f6b7c00cc60 comp rx=0 tx=0).stop 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 shutdown_connections 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6b64076480 0x7f6b64078940 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b8415a6b0 0x7f6b841f3d00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b840cc1c0 0x7f6b8415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 --2- 172.21.15.163:0/3304102622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b840741f0 0x7f6b8414f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 >> 172.21.15.163:0/3304102622 conn(0x7f6b840b0e30 msgr2=0x7f6b84159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 shutdown_connections 2024-07-26T06:15:14.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.335+0000 7f6b8b000640 1 -- 172.21.15.163:0/3304102622 wait complete. 2024-07-26T06:15:14.356 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-07-26T06:15:13.678704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225616,"kb_used_data":5288,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749104,"statfs":{"total":773094113280,"available":772863082496,"internally_reserved":0,"allocated":5414912,"data_stored":2871196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.284852"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429503+0000","last_change":"2024-07-26T06:15:07.439002+0000","last_active":"2024-07-26T06:15:08.429503+0000","last_peered":"2024-07-26T06:15:08.429503+0000","last_clean":"2024-07-26T06:15:08.429503+0000","last_became_active":"2024-07-26T06:15:07.437642+0000","last_became_peered":"2024-07-26T06:15:07.437642+0000","last_unstale":"2024-07-26T06:15:08.429503+0000","last_undegraded":"2024-07-26T06:15:08.429503+0000","last_fullsized":"2024-07-26T06:15:08.429503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:37.698071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165913+0000","last_change":"2024-07-26T06:15:07.443943+0000","last_active":"2024-07-26T06:15:10.165913+0000","last_peered":"2024-07-26T06:15:10.165913+0000","last_clean":"2024-07-26T06:15:10.165913+0000","last_became_active":"2024-07-26T06:15:07.443630+0000","last_became_peered":"2024-07-26T06:15:07.443630+0000","last_unstale":"2024-07-26T06:15:10.165913+0000","last_undegraded":"2024-07-26T06:15:10.165913+0000","last_fullsized":"2024-07-26T06:15:10.165913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:04.663181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442388+0000","last_change":"2024-07-26T06:15:07.428380+0000","last_active":"2024-07-26T06:15:08.442388+0000","last_peered":"2024-07-26T06:15:08.442388+0000","last_clean":"2024-07-26T06:15:08.442388+0000","last_became_active":"2024-07-26T06:15:07.428033+0000","last_became_peered":"2024-07-26T06:15:07.428033+0000","last_unstale":"2024-07-26T06:15:08.442388+0000","last_undegraded":"2024-07-26T06:15:08.442388+0000","last_fullsized":"2024-07-26T06:15:08.442388+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:57:29.467958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427784+0000","last_change":"2024-07-26T06:15:07.446508+0000","last_active":"2024-07-26T06:15:08.427784+0000","last_peered":"2024-07-26T06:15:08.427784+0000","last_clean":"2024-07-26T06:15:08.427784+0000","last_became_active":"2024-07-26T06:15:07.446277+0000","last_became_peered":"2024-07-26T06:15:07.446277+0000","last_unstale":"2024-07-26T06:15:08.427784+0000","last_undegraded":"2024-07-26T06:15:08.427784+0000","last_fullsized":"2024-07-26T06:15:08.427784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:56.749715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165115+0000","last_change":"2024-07-26T06:15:07.442881+0000","last_active":"2024-07-26T06:15:10.165115+0000","last_peered":"2024-07-26T06:15:10.165115+0000","last_clean":"2024-07-26T06:15:10.165115+0000","last_became_active":"2024-07-26T06:15:07.442563+0000","last_became_peered":"2024-07-26T06:15:07.442563+0000","last_unstale":"2024-07-26T06:15:10.165115+0000","last_undegraded":"2024-07-26T06:15:10.165115+0000","last_fullsized":"2024-07-26T06:15:10.165115+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:38.427858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431092+0000","last_change":"2024-07-26T06:15:07.439882+0000","last_active":"2024-07-26T06:15:08.431092+0000","last_peered":"2024-07-26T06:15:08.431092+0000","last_clean":"2024-07-26T06:15:08.431092+0000","last_became_active":"2024-07-26T06:15:07.439317+0000","last_became_peered":"2024-07-26T06:15:07.439317+0000","last_unstale":"2024-07-26T06:15:08.431092+0000","last_undegraded":"2024-07-26T06:15:08.431092+0000","last_fullsized":"2024-07-26T06:15:08.431092+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:22:51.472360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428232+0000","last_change":"2024-07-26T06:15:07.441524+0000","last_active":"2024-07-26T06:15:08.428232+0000","last_peered":"2024-07-26T06:15:08.428232+0000","last_clean":"2024-07-26T06:15:08.428232+0000","last_became_active":"2024-07-26T06:15:07.441230+0000","last_became_peered":"2024-07-26T06:15:07.441230+0000","last_unstale":"2024-07-26T06:15:08.428232+0000","last_undegraded":"2024-07-26T06:15:08.428232+0000","last_fullsized":"2024-07-26T06:15:08.428232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:54:04.663181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427054+0000","last_change":"2024-07-26T06:15:07.456220+0000","last_active":"2024-07-26T06:15:08.427054+0000","last_peered":"2024-07-26T06:15:08.427054+0000","last_clean":"2024-07-26T06:15:08.427054+0000","last_became_active":"2024-07-26T06:15:07.455408+0000","last_became_peered":"2024-07-26T06:15:07.455408+0000","last_unstale":"2024-07-26T06:15:08.427054+0000","last_undegraded":"2024-07-26T06:15:08.427054+0000","last_fullsized":"2024-07-26T06:15:08.427054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:16:49.612701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430334+0000","last_change":"2024-07-26T06:15:07.430412+0000","last_active":"2024-07-26T06:15:08.430334+0000","last_peered":"2024-07-26T06:15:08.430334+0000","last_clean":"2024-07-26T06:15:08.430334+0000","last_became_active":"2024-07-26T06:15:07.430114+0000","last_became_peered":"2024-07-26T06:15:07.430114+0000","last_unstale":"2024-07-26T06:15:08.430334+0000","last_undegraded":"2024-07-26T06:15:08.430334+0000","last_fullsized":"2024-07-26T06:15:08.430334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:38.427858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430061+0000","last_change":"2024-07-26T06:15:07.454615+0000","last_active":"2024-07-26T06:15:08.430061+0000","last_peered":"2024-07-26T06:15:08.430061+0000","last_clean":"2024-07-26T06:15:08.430061+0000","last_became_active":"2024-07-26T06:15:07.454233+0000","last_became_peered":"2024-07-26T06:15:07.454233+0000","last_unstale":"2024-07-26T06:15:08.430061+0000","last_undegraded":"2024-07-26T06:15:08.430061+0000","last_fullsized":"2024-07-26T06:15:08.430061+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:55.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429430+0000","last_change":"2024-07-26T06:15:07.428324+0000","last_active":"2024-07-26T06:15:08.429430+0000","last_peered":"2024-07-26T06:15:08.429430+0000","last_clean":"2024-07-26T06:15:08.429430+0000","last_became_active":"2024-07-26T06:15:07.428005+0000","last_became_peered":"2024-07-26T06:15:07.428005+0000","last_unstale":"2024-07-26T06:15:08.429430+0000","last_undegraded":"2024-07-26T06:15:08.429430+0000","last_fullsized":"2024-07-26T06:15:08.429430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:49.119077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165498+0000","last_change":"2024-07-26T06:15:07.440834+0000","last_active":"2024-07-26T06:15:10.165498+0000","last_peered":"2024-07-26T06:15:10.165498+0000","last_clean":"2024-07-26T06:15:10.165498+0000","last_became_active":"2024-07-26T06:15:07.440254+0000","last_became_peered":"2024-07-26T06:15:07.440254+0000","last_unstale":"2024-07-26T06:15:10.165498+0000","last_undegraded":"2024-07-26T06:15:10.165498+0000","last_fullsized":"2024-07-26T06:15:10.165498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:44.629861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429826+0000","last_change":"2024-07-26T06:15:07.434318+0000","last_active":"2024-07-26T06:15:08.429826+0000","last_peered":"2024-07-26T06:15:08.429826+0000","last_clean":"2024-07-26T06:15:08.429826+0000","last_became_active":"2024-07-26T06:15:07.434035+0000","last_became_peered":"2024-07-26T06:15:07.434035+0000","last_unstale":"2024-07-26T06:15:08.429826+0000","last_undegraded":"2024-07-26T06:15:08.429826+0000","last_fullsized":"2024-07-26T06:15:08.429826+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:32.612822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427745+0000","last_change":"2024-07-26T06:15:07.428831+0000","last_active":"2024-07-26T06:15:08.427745+0000","last_peered":"2024-07-26T06:15:08.427745+0000","last_clean":"2024-07-26T06:15:08.427745+0000","last_became_active":"2024-07-26T06:15:07.428547+0000","last_became_peered":"2024-07-26T06:15:07.428547+0000","last_unstale":"2024-07-26T06:15:08.427745+0000","last_undegraded":"2024-07-26T06:15:08.427745+0000","last_fullsized":"2024-07-26T06:15:08.427745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:35.122591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431216+0000","last_change":"2024-07-26T06:15:07.421441+0000","last_active":"2024-07-26T06:15:08.431216+0000","last_peered":"2024-07-26T06:15:08.431216+0000","last_clean":"2024-07-26T06:15:08.431216+0000","last_became_active":"2024-07-26T06:15:07.421000+0000","last_became_peered":"2024-07-26T06:15:07.421000+0000","last_unstale":"2024-07-26T06:15:08.431216+0000","last_undegraded":"2024-07-26T06:15:08.431216+0000","last_fullsized":"2024-07-26T06:15:08.431216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:35:48.576971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063818+0000","last_change":"2024-07-26T06:15:07.447544+0000","last_active":"2024-07-26T06:15:09.063818+0000","last_peered":"2024-07-26T06:15:09.063818+0000","last_clean":"2024-07-26T06:15:09.063818+0000","last_became_active":"2024-07-26T06:15:07.447200+0000","last_became_peered":"2024-07-26T06:15:07.447200+0000","last_unstale":"2024-07-26T06:15:09.063818+0000","last_undegraded":"2024-07-26T06:15:09.063818+0000","last_fullsized":"2024-07-26T06:15:09.063818+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:24.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442761+0000","last_change":"2024-07-26T06:15:07.433146+0000","last_active":"2024-07-26T06:15:08.442761+0000","last_peered":"2024-07-26T06:15:08.442761+0000","last_clean":"2024-07-26T06:15:08.442761+0000","last_became_active":"2024-07-26T06:15:07.432583+0000","last_became_peered":"2024-07-26T06:15:07.432583+0000","last_unstale":"2024-07-26T06:15:08.442761+0000","last_undegraded":"2024-07-26T06:15:08.442761+0000","last_fullsized":"2024-07-26T06:15:08.442761+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:11.453191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442768+0000","last_change":"2024-07-26T06:15:07.431772+0000","last_active":"2024-07-26T06:15:08.442768+0000","last_peered":"2024-07-26T06:15:08.442768+0000","last_clean":"2024-07-26T06:15:08.442768+0000","last_became_active":"2024-07-26T06:15:07.430990+0000","last_became_peered":"2024-07-26T06:15:07.430990+0000","last_unstale":"2024-07-26T06:15:08.442768+0000","last_undegraded":"2024-07-26T06:15:08.442768+0000","last_fullsized":"2024-07-26T06:15:08.442768+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:19:53.598400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063547+0000","last_change":"2024-07-26T06:15:07.446491+0000","last_active":"2024-07-26T06:15:09.063547+0000","last_peered":"2024-07-26T06:15:09.063547+0000","last_clean":"2024-07-26T06:15:09.063547+0000","last_became_active":"2024-07-26T06:15:07.446167+0000","last_became_peered":"2024-07-26T06:15:07.446167+0000","last_unstale":"2024-07-26T06:15:09.063547+0000","last_undegraded":"2024-07-26T06:15:09.063547+0000","last_fullsized":"2024-07-26T06:15:09.063547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:00:47.313017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.433244+0000","last_change":"2024-07-26T06:15:07.430059+0000","last_active":"2024-07-26T06:15:08.433244+0000","last_peered":"2024-07-26T06:15:08.433244+0000","last_clean":"2024-07-26T06:15:08.433244+0000","last_became_active":"2024-07-26T06:15:07.429271+0000","last_became_peered":"2024-07-26T06:15:07.429271+0000","last_unstale":"2024-07-26T06:15:08.433244+0000","last_undegraded":"2024-07-26T06:15:08.433244+0000","last_fullsized":"2024-07-26T06:15:08.433244+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:38.518133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431704+0000","last_change":"2024-07-26T06:15:07.447086+0000","last_active":"2024-07-26T06:15:08.431704+0000","last_peered":"2024-07-26T06:15:08.431704+0000","last_clean":"2024-07-26T06:15:08.431704+0000","last_became_active":"2024-07-26T06:15:07.446851+0000","last_became_peered":"2024-07-26T06:15:07.446851+0000","last_unstale":"2024-07-26T06:15:08.431704+0000","last_undegraded":"2024-07-26T06:15:08.431704+0000","last_fullsized":"2024-07-26T06:15:08.431704+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:34:53.383778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428183+0000","last_change":"2024-07-26T06:15:07.455293+0000","last_active":"2024-07-26T06:15:08.428183+0000","last_peered":"2024-07-26T06:15:08.428183+0000","last_clean":"2024-07-26T06:15:08.428183+0000","last_became_active":"2024-07-26T06:15:07.454786+0000","last_became_peered":"2024-07-26T06:15:07.454786+0000","last_unstale":"2024-07-26T06:15:08.428183+0000","last_undegraded":"2024-07-26T06:15:08.428183+0000","last_fullsized":"2024-07-26T06:15:08.428183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:24:58.110547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063714+0000","last_change":"2024-07-26T06:15:07.448843+0000","last_active":"2024-07-26T06:15:09.063714+0000","last_peered":"2024-07-26T06:15:09.063714+0000","last_clean":"2024-07-26T06:15:09.063714+0000","last_became_active":"2024-07-26T06:15:07.448567+0000","last_became_peered":"2024-07-26T06:15:07.448567+0000","last_unstale":"2024-07-26T06:15:09.063714+0000","last_undegraded":"2024-07-26T06:15:09.063714+0000","last_fullsized":"2024-07-26T06:15:09.063714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:22.751385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063090+0000","last_change":"2024-07-26T06:15:07.444695+0000","last_active":"2024-07-26T06:15:09.063090+0000","last_peered":"2024-07-26T06:15:09.063090+0000","last_clean":"2024-07-26T06:15:09.063090+0000","last_became_active":"2024-07-26T06:15:07.444418+0000","last_became_peered":"2024-07-26T06:15:07.444418+0000","last_unstale":"2024-07-26T06:15:09.063090+0000","last_undegraded":"2024-07-26T06:15:09.063090+0000","last_fullsized":"2024-07-26T06:15:09.063090+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:20.118927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063436+0000","last_change":"2024-07-26T06:15:07.454543+0000","last_active":"2024-07-26T06:15:09.063436+0000","last_peered":"2024-07-26T06:15:09.063436+0000","last_clean":"2024-07-26T06:15:09.063436+0000","last_became_active":"2024-07-26T06:15:07.452159+0000","last_became_peered":"2024-07-26T06:15:07.452159+0000","last_unstale":"2024-07-26T06:15:09.063436+0000","last_undegraded":"2024-07-26T06:15:09.063436+0000","last_fullsized":"2024-07-26T06:15:09.063436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:35:53.361444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430097+0000","last_change":"2024-07-26T06:15:07.434897+0000","last_active":"2024-07-26T06:15:08.430097+0000","last_peered":"2024-07-26T06:15:08.430097+0000","last_clean":"2024-07-26T06:15:08.430097+0000","last_became_active":"2024-07-26T06:15:07.434456+0000","last_became_peered":"2024-07-26T06:15:07.434456+0000","last_unstale":"2024-07-26T06:15:08.430097+0000","last_undegraded":"2024-07-26T06:15:08.430097+0000","last_fullsized":"2024-07-26T06:15:08.430097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:07:44.757628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442528+0000","last_change":"2024-07-26T06:15:07.442852+0000","last_active":"2024-07-26T06:15:08.442528+0000","last_peered":"2024-07-26T06:15:08.442528+0000","last_clean":"2024-07-26T06:15:08.442528+0000","last_became_active":"2024-07-26T06:15:07.440917+0000","last_became_peered":"2024-07-26T06:15:07.440917+0000","last_unstale":"2024-07-26T06:15:08.442528+0000","last_undegraded":"2024-07-26T06:15:08.442528+0000","last_fullsized":"2024-07-26T06:15:08.442528+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:33.985738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165675+0000","last_change":"2024-07-26T06:15:07.440581+0000","last_active":"2024-07-26T06:15:10.165675+0000","last_peered":"2024-07-26T06:15:10.165675+0000","last_clean":"2024-07-26T06:15:10.165675+0000","last_became_active":"2024-07-26T06:15:07.439909+0000","last_became_peered":"2024-07-26T06:15:07.439909+0000","last_unstale":"2024-07-26T06:15:10.165675+0000","last_undegraded":"2024-07-26T06:15:10.165675+0000","last_fullsized":"2024-07-26T06:15:10.165675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:21:23.009970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063424+0000","last_change":"2024-07-26T06:15:07.452847+0000","last_active":"2024-07-26T06:15:09.063424+0000","last_peered":"2024-07-26T06:15:09.063424+0000","last_clean":"2024-07-26T06:15:09.063424+0000","last_became_active":"2024-07-26T06:15:07.452431+0000","last_became_peered":"2024-07-26T06:15:07.452431+0000","last_unstale":"2024-07-26T06:15:09.063424+0000","last_undegraded":"2024-07-26T06:15:09.063424+0000","last_fullsized":"2024-07-26T06:15:09.063424+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:14:36.028736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442460+0000","last_change":"2024-07-26T06:15:07.438387+0000","last_active":"2024-07-26T06:15:08.442460+0000","last_peered":"2024-07-26T06:15:08.442460+0000","last_clean":"2024-07-26T06:15:08.442460+0000","last_became_active":"2024-07-26T06:15:07.437984+0000","last_became_peered":"2024-07-26T06:15:07.437984+0000","last_unstale":"2024-07-26T06:15:08.442460+0000","last_undegraded":"2024-07-26T06:15:08.442460+0000","last_fullsized":"2024-07-26T06:15:08.442460+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:42:51.944062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432952+0000","last_change":"2024-07-26T06:15:07.435958+0000","last_active":"2024-07-26T06:15:08.432952+0000","last_peered":"2024-07-26T06:15:08.432952+0000","last_clean":"2024-07-26T06:15:08.432952+0000","last_became_active":"2024-07-26T06:15:07.435636+0000","last_became_peered":"2024-07-26T06:15:07.435636+0000","last_unstale":"2024-07-26T06:15:08.432952+0000","last_undegraded":"2024-07-26T06:15:08.432952+0000","last_fullsized":"2024-07-26T06:15:08.432952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:09:06.481920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427528+0000","last_change":"2024-07-26T06:15:07.429783+0000","last_active":"2024-07-26T06:15:08.427528+0000","last_peered":"2024-07-26T06:15:08.427528+0000","last_clean":"2024-07-26T06:15:08.427528+0000","last_became_active":"2024-07-26T06:15:07.429434+0000","last_became_peered":"2024-07-26T06:15:07.429434+0000","last_unstale":"2024-07-26T06:15:08.427528+0000","last_undegraded":"2024-07-26T06:15:08.427528+0000","last_fullsized":"2024-07-26T06:15:08.427528+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:55.220456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162612+0000","last_change":"2024-07-26T06:15:07.428785+0000","last_active":"2024-07-26T06:15:10.162612+0000","last_peered":"2024-07-26T06:15:10.162612+0000","last_clean":"2024-07-26T06:15:10.162612+0000","last_became_active":"2024-07-26T06:15:07.428493+0000","last_became_peered":"2024-07-26T06:15:07.428493+0000","last_unstale":"2024-07-26T06:15:10.162612+0000","last_undegraded":"2024-07-26T06:15:10.162612+0000","last_fullsized":"2024-07-26T06:15:10.162612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:00.268991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432740+0000","last_change":"2024-07-26T06:15:06.428719+0000","last_active":"2024-07-26T06:15:08.432740+0000","last_peered":"2024-07-26T06:15:08.432740+0000","last_clean":"2024-07-26T06:15:08.432740+0000","last_became_active":"2024-07-26T06:15:06.428416+0000","last_became_peered":"2024-07-26T06:15:06.428416+0000","last_unstale":"2024-07-26T06:15:08.432740+0000","last_undegraded":"2024-07-26T06:15:08.432740+0000","last_fullsized":"2024-07-26T06:15:08.432740+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:25.572744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063722+0000","last_change":"2024-07-26T06:15:07.447042+0000","last_active":"2024-07-26T06:15:09.063722+0000","last_peered":"2024-07-26T06:15:09.063722+0000","last_clean":"2024-07-26T06:15:09.063722+0000","last_became_active":"2024-07-26T06:15:07.446737+0000","last_became_peered":"2024-07-26T06:15:07.446737+0000","last_unstale":"2024-07-26T06:15:09.063722+0000","last_undegraded":"2024-07-26T06:15:09.063722+0000","last_fullsized":"2024-07-26T06:15:09.063722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:17:34.177718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442883+0000","last_change":"2024-07-26T06:15:06.425064+0000","last_active":"2024-07-26T06:15:08.442883+0000","last_peered":"2024-07-26T06:15:08.442883+0000","last_clean":"2024-07-26T06:15:08.442883+0000","last_became_active":"2024-07-26T06:15:06.424560+0000","last_became_peered":"2024-07-26T06:15:06.424560+0000","last_unstale":"2024-07-26T06:15:08.442883+0000","last_undegraded":"2024-07-26T06:15:08.442883+0000","last_fullsized":"2024-07-26T06:15:08.442883+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:32:52.880734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165852+0000","last_change":"2024-07-26T06:15:07.438595+0000","last_active":"2024-07-26T06:15:10.165852+0000","last_peered":"2024-07-26T06:15:10.165852+0000","last_clean":"2024-07-26T06:15:10.165852+0000","last_became_active":"2024-07-26T06:15:07.438207+0000","last_became_peered":"2024-07-26T06:15:07.438207+0000","last_unstale":"2024-07-26T06:15:10.165852+0000","last_undegraded":"2024-07-26T06:15:10.165852+0000","last_fullsized":"2024-07-26T06:15:10.165852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:30:23.809092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162857+0000","last_change":"2024-07-26T06:15:06.428424+0000","last_active":"2024-07-26T06:15:10.162857+0000","last_peered":"2024-07-26T06:15:10.162857+0000","last_clean":"2024-07-26T06:15:10.162857+0000","last_became_active":"2024-07-26T06:15:06.428100+0000","last_became_peered":"2024-07-26T06:15:06.428100+0000","last_unstale":"2024-07-26T06:15:10.162857+0000","last_undegraded":"2024-07-26T06:15:10.162857+0000","last_fullsized":"2024-07-26T06:15:10.162857+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:32:52.880734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.163002+0000","last_change":"2024-07-26T06:15:07.456057+0000","last_active":"2024-07-26T06:15:10.163002+0000","last_peered":"2024-07-26T06:15:10.163002+0000","last_clean":"2024-07-26T06:15:10.163002+0000","last_became_active":"2024-07-26T06:15:07.455730+0000","last_became_peered":"2024-07-26T06:15:07.455730+0000","last_unstale":"2024-07-26T06:15:10.163002+0000","last_undegraded":"2024-07-26T06:15:10.163002+0000","last_fullsized":"2024-07-26T06:15:10.163002+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:18:56.750715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428991+0000","last_change":"2024-07-26T06:15:06.434477+0000","last_active":"2024-07-26T06:15:08.428991+0000","last_peered":"2024-07-26T06:15:08.428991+0000","last_clean":"2024-07-26T06:15:08.428991+0000","last_became_active":"2024-07-26T06:15:06.433902+0000","last_became_peered":"2024-07-26T06:15:06.433902+0000","last_unstale":"2024-07-26T06:15:08.428991+0000","last_undegraded":"2024-07-26T06:15:08.428991+0000","last_fullsized":"2024-07-26T06:15:08.428991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:24:57.104664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442227+0000","last_change":"2024-07-26T06:15:07.446994+0000","last_active":"2024-07-26T06:15:08.442227+0000","last_peered":"2024-07-26T06:15:08.442227+0000","last_clean":"2024-07-26T06:15:08.442227+0000","last_became_active":"2024-07-26T06:15:07.446724+0000","last_became_peered":"2024-07-26T06:15:07.446724+0000","last_unstale":"2024-07-26T06:15:08.442227+0000","last_undegraded":"2024-07-26T06:15:08.442227+0000","last_fullsized":"2024-07-26T06:15:08.442227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:00.713979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431870+0000","last_change":"2024-07-26T06:15:06.430887+0000","last_active":"2024-07-26T06:15:08.431870+0000","last_peered":"2024-07-26T06:15:08.431870+0000","last_clean":"2024-07-26T06:15:08.431870+0000","last_became_active":"2024-07-26T06:15:06.429962+0000","last_became_peered":"2024-07-26T06:15:06.429962+0000","last_unstale":"2024-07-26T06:15:08.431870+0000","last_undegraded":"2024-07-26T06:15:08.431870+0000","last_fullsized":"2024-07-26T06:15:08.431870+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:54:06.599271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428762+0000","last_change":"2024-07-26T06:15:07.432241+0000","last_active":"2024-07-26T06:15:08.428762+0000","last_peered":"2024-07-26T06:15:08.428762+0000","last_clean":"2024-07-26T06:15:08.428762+0000","last_became_active":"2024-07-26T06:15:07.431906+0000","last_became_peered":"2024-07-26T06:15:07.431906+0000","last_unstale":"2024-07-26T06:15:08.428762+0000","last_undegraded":"2024-07-26T06:15:08.428762+0000","last_fullsized":"2024-07-26T06:15:08.428762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:09.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442398+0000","last_change":"2024-07-26T06:15:06.431877+0000","last_active":"2024-07-26T06:15:08.442398+0000","last_peered":"2024-07-26T06:15:08.442398+0000","last_clean":"2024-07-26T06:15:08.442398+0000","last_became_active":"2024-07-26T06:15:06.431309+0000","last_became_peered":"2024-07-26T06:15:06.431309+0000","last_unstale":"2024-07-26T06:15:08.442398+0000","last_undegraded":"2024-07-26T06:15:08.442398+0000","last_fullsized":"2024-07-26T06:15:08.442398+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:03:46.856382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427311+0000","last_change":"2024-07-26T06:15:07.446522+0000","last_active":"2024-07-26T06:15:08.427311+0000","last_peered":"2024-07-26T06:15:08.427311+0000","last_clean":"2024-07-26T06:15:08.427311+0000","last_became_active":"2024-07-26T06:15:07.446199+0000","last_became_peered":"2024-07-26T06:15:07.446199+0000","last_unstale":"2024-07-26T06:15:08.427311+0000","last_undegraded":"2024-07-26T06:15:08.427311+0000","last_fullsized":"2024-07-26T06:15:08.427311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:41.575875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165173+0000","last_change":"2024-07-26T06:15:06.428513+0000","last_active":"2024-07-26T06:15:10.165173+0000","last_peered":"2024-07-26T06:15:10.165173+0000","last_clean":"2024-07-26T06:15:10.165173+0000","last_became_active":"2024-07-26T06:15:06.427510+0000","last_became_peered":"2024-07-26T06:15:06.427510+0000","last_unstale":"2024-07-26T06:15:10.165173+0000","last_undegraded":"2024-07-26T06:15:10.165173+0000","last_fullsized":"2024-07-26T06:15:10.165173+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:34.116708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063181+0000","last_change":"2024-07-26T06:15:07.453292+0000","last_active":"2024-07-26T06:15:09.063181+0000","last_peered":"2024-07-26T06:15:09.063181+0000","last_clean":"2024-07-26T06:15:09.063181+0000","last_became_active":"2024-07-26T06:15:07.451899+0000","last_became_peered":"2024-07-26T06:15:07.451899+0000","last_unstale":"2024-07-26T06:15:09.063181+0000","last_undegraded":"2024-07-26T06:15:09.063181+0000","last_fullsized":"2024-07-26T06:15:09.063181+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:05.145115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427367+0000","last_change":"2024-07-26T06:15:06.416806+0000","last_active":"2024-07-26T06:15:08.427367+0000","last_peered":"2024-07-26T06:15:08.427367+0000","last_clean":"2024-07-26T06:15:08.427367+0000","last_became_active":"2024-07-26T06:15:06.415836+0000","last_became_peered":"2024-07-26T06:15:06.415836+0000","last_unstale":"2024-07-26T06:15:08.427367+0000","last_undegraded":"2024-07-26T06:15:08.427367+0000","last_fullsized":"2024-07-26T06:15:08.427367+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:44:54.680418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442955+0000","last_change":"2024-07-26T06:15:07.427313+0000","last_active":"2024-07-26T06:15:08.442955+0000","last_peered":"2024-07-26T06:15:08.442955+0000","last_clean":"2024-07-26T06:15:08.442955+0000","last_became_active":"2024-07-26T06:15:07.427007+0000","last_became_peered":"2024-07-26T06:15:07.427007+0000","last_unstale":"2024-07-26T06:15:08.442955+0000","last_undegraded":"2024-07-26T06:15:08.442955+0000","last_fullsized":"2024-07-26T06:15:08.442955+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:26.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442902+0000","last_change":"2024-07-26T06:15:06.418583+0000","last_active":"2024-07-26T06:15:08.442902+0000","last_peered":"2024-07-26T06:15:08.442902+0000","last_clean":"2024-07-26T06:15:08.442902+0000","last_became_active":"2024-07-26T06:15:06.417671+0000","last_became_peered":"2024-07-26T06:15:06.417671+0000","last_unstale":"2024-07-26T06:15:08.442902+0000","last_undegraded":"2024-07-26T06:15:08.442902+0000","last_fullsized":"2024-07-26T06:15:08.442902+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:58:07.482287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429547+0000","last_change":"2024-07-26T06:15:07.449855+0000","last_active":"2024-07-26T06:15:08.429547+0000","last_peered":"2024-07-26T06:15:08.429547+0000","last_clean":"2024-07-26T06:15:08.429547+0000","last_became_active":"2024-07-26T06:15:07.449557+0000","last_became_peered":"2024-07-26T06:15:07.449557+0000","last_unstale":"2024-07-26T06:15:08.429547+0000","last_undegraded":"2024-07-26T06:15:08.429547+0000","last_fullsized":"2024-07-26T06:15:08.429547+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:01:48.971591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430340+0000","last_change":"2024-07-26T06:15:06.427881+0000","last_active":"2024-07-26T06:15:08.430340+0000","last_peered":"2024-07-26T06:15:08.430340+0000","last_clean":"2024-07-26T06:15:08.430340+0000","last_became_active":"2024-07-26T06:15:06.427244+0000","last_became_peered":"2024-07-26T06:15:06.427244+0000","last_unstale":"2024-07-26T06:15:08.430340+0000","last_undegraded":"2024-07-26T06:15:08.430340+0000","last_fullsized":"2024-07-26T06:15:08.430340+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:31.606939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432752+0000","last_change":"2024-07-26T06:15:07.433401+0000","last_active":"2024-07-26T06:15:08.432752+0000","last_peered":"2024-07-26T06:15:08.432752+0000","last_clean":"2024-07-26T06:15:08.432752+0000","last_became_active":"2024-07-26T06:15:07.432643+0000","last_became_peered":"2024-07-26T06:15:07.432643+0000","last_unstale":"2024-07-26T06:15:08.432752+0000","last_undegraded":"2024-07-26T06:15:08.432752+0000","last_fullsized":"2024-07-26T06:15:08.432752+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:09.016877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429480+0000","last_change":"2024-07-26T06:15:06.440211+0000","last_active":"2024-07-26T06:15:08.429480+0000","last_peered":"2024-07-26T06:15:08.429480+0000","last_clean":"2024-07-26T06:15:08.429480+0000","last_became_active":"2024-07-26T06:15:06.439822+0000","last_became_peered":"2024-07-26T06:15:06.439822+0000","last_unstale":"2024-07-26T06:15:08.429480+0000","last_undegraded":"2024-07-26T06:15:08.429480+0000","last_fullsized":"2024-07-26T06:15:08.429480+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:47:08.567851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429209+0000","last_change":"2024-07-26T06:15:07.441660+0000","last_active":"2024-07-26T06:15:08.429209+0000","last_peered":"2024-07-26T06:15:08.429209+0000","last_clean":"2024-07-26T06:15:08.429209+0000","last_became_active":"2024-07-26T06:15:07.441406+0000","last_became_peered":"2024-07-26T06:15:07.441406+0000","last_unstale":"2024-07-26T06:15:08.429209+0000","last_undegraded":"2024-07-26T06:15:08.429209+0000","last_fullsized":"2024-07-26T06:15:08.429209+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:03:35.230183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431920+0000","last_change":"2024-07-26T06:15:06.421856+0000","last_active":"2024-07-26T06:15:08.431920+0000","last_peered":"2024-07-26T06:15:08.431920+0000","last_clean":"2024-07-26T06:15:08.431920+0000","last_became_active":"2024-07-26T06:15:06.421246+0000","last_became_peered":"2024-07-26T06:15:06.421246+0000","last_unstale":"2024-07-26T06:15:08.431920+0000","last_undegraded":"2024-07-26T06:15:08.431920+0000","last_fullsized":"2024-07-26T06:15:08.431920+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:21.906289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063131+0000","last_change":"2024-07-26T06:15:07.453126+0000","last_active":"2024-07-26T06:15:09.063131+0000","last_peered":"2024-07-26T06:15:09.063131+0000","last_clean":"2024-07-26T06:15:09.063131+0000","last_became_active":"2024-07-26T06:15:07.451881+0000","last_became_peered":"2024-07-26T06:15:07.451881+0000","last_unstale":"2024-07-26T06:15:09.063131+0000","last_undegraded":"2024-07-26T06:15:09.063131+0000","last_fullsized":"2024-07-26T06:15:09.063131+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:03.327048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432445+0000","last_change":"2024-07-26T06:15:06.425695+0000","last_active":"2024-07-26T06:15:08.432445+0000","last_peered":"2024-07-26T06:15:08.432445+0000","last_clean":"2024-07-26T06:15:08.432445+0000","last_became_active":"2024-07-26T06:15:06.425385+0000","last_became_peered":"2024-07-26T06:15:06.425385+0000","last_unstale":"2024-07-26T06:15:08.432445+0000","last_undegraded":"2024-07-26T06:15:08.432445+0000","last_fullsized":"2024-07-26T06:15:08.432445+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:50:25.392213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429021+0000","last_change":"2024-07-26T06:15:07.440623+0000","last_active":"2024-07-26T06:15:08.429021+0000","last_peered":"2024-07-26T06:15:08.429021+0000","last_clean":"2024-07-26T06:15:08.429021+0000","last_became_active":"2024-07-26T06:15:07.440297+0000","last_became_peered":"2024-07-26T06:15:07.440297+0000","last_unstale":"2024-07-26T06:15:08.429021+0000","last_undegraded":"2024-07-26T06:15:08.429021+0000","last_fullsized":"2024-07-26T06:15:08.429021+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:51.343615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428657+0000","last_change":"2024-07-26T06:15:06.430182+0000","last_active":"2024-07-26T06:15:08.428657+0000","last_peered":"2024-07-26T06:15:08.428657+0000","last_clean":"2024-07-26T06:15:08.428657+0000","last_became_active":"2024-07-26T06:15:06.429864+0000","last_became_peered":"2024-07-26T06:15:06.429864+0000","last_unstale":"2024-07-26T06:15:08.428657+0000","last_undegraded":"2024-07-26T06:15:08.428657+0000","last_fullsized":"2024-07-26T06:15:08.428657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:06:34.116708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431100+0000","last_change":"2024-07-26T06:15:07.432954+0000","last_active":"2024-07-26T06:15:08.431100+0000","last_peered":"2024-07-26T06:15:08.431100+0000","last_clean":"2024-07-26T06:15:08.431100+0000","last_became_active":"2024-07-26T06:15:07.432654+0000","last_became_peered":"2024-07-26T06:15:07.432654+0000","last_unstale":"2024-07-26T06:15:08.431100+0000","last_undegraded":"2024-07-26T06:15:08.431100+0000","last_fullsized":"2024-07-26T06:15:08.431100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:46:09.745514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432280+0000","last_change":"2024-07-26T06:15:06.425752+0000","last_active":"2024-07-26T06:15:08.432280+0000","last_peered":"2024-07-26T06:15:08.432280+0000","last_clean":"2024-07-26T06:15:08.432280+0000","last_became_active":"2024-07-26T06:15:06.425005+0000","last_became_peered":"2024-07-26T06:15:06.425005+0000","last_unstale":"2024-07-26T06:15:08.432280+0000","last_undegraded":"2024-07-26T06:15:08.432280+0000","last_fullsized":"2024-07-26T06:15:08.432280+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:36:12.256587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428351+0000","last_change":"2024-07-26T06:15:07.457696+0000","last_active":"2024-07-26T06:15:08.428351+0000","last_peered":"2024-07-26T06:15:08.428351+0000","last_clean":"2024-07-26T06:15:08.428351+0000","last_became_active":"2024-07-26T06:15:07.454380+0000","last_became_peered":"2024-07-26T06:15:07.454380+0000","last_unstale":"2024-07-26T06:15:08.428351+0000","last_undegraded":"2024-07-26T06:15:08.428351+0000","last_fullsized":"2024-07-26T06:15:08.428351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:21:23.009970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427661+0000","last_change":"2024-07-26T06:15:06.427250+0000","last_active":"2024-07-26T06:15:08.427661+0000","last_peered":"2024-07-26T06:15:08.427661+0000","last_clean":"2024-07-26T06:15:08.427661+0000","last_became_active":"2024-07-26T06:15:06.426832+0000","last_became_peered":"2024-07-26T06:15:06.426832+0000","last_unstale":"2024-07-26T06:15:08.427661+0000","last_undegraded":"2024-07-26T06:15:08.427661+0000","last_fullsized":"2024-07-26T06:15:08.427661+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:54.214574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430516+0000","last_change":"2024-07-26T06:15:07.442796+0000","last_active":"2024-07-26T06:15:08.430516+0000","last_peered":"2024-07-26T06:15:08.430516+0000","last_clean":"2024-07-26T06:15:08.430516+0000","last_became_active":"2024-07-26T06:15:07.440503+0000","last_became_peered":"2024-07-26T06:15:07.440503+0000","last_unstale":"2024-07-26T06:15:08.430516+0000","last_undegraded":"2024-07-26T06:15:08.430516+0000","last_fullsized":"2024-07-26T06:15:08.430516+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:17.798570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.433106+0000","last_change":"2024-07-26T06:15:06.421238+0000","last_active":"2024-07-26T06:15:08.433106+0000","last_peered":"2024-07-26T06:15:08.433106+0000","last_clean":"2024-07-26T06:15:08.433106+0000","last_became_active":"2024-07-26T06:15:06.420892+0000","last_became_peered":"2024-07-26T06:15:06.420892+0000","last_unstale":"2024-07-26T06:15:08.433106+0000","last_undegraded":"2024-07-26T06:15:08.433106+0000","last_fullsized":"2024-07-26T06:15:08.433106+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:39.822740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430381+0000","last_change":"2024-07-26T06:15:07.421052+0000","last_active":"2024-07-26T06:15:08.430381+0000","last_peered":"2024-07-26T06:15:08.430381+0000","last_clean":"2024-07-26T06:15:08.430381+0000","last_became_active":"2024-07-26T06:15:07.420689+0000","last_became_peered":"2024-07-26T06:15:07.420689+0000","last_unstale":"2024-07-26T06:15:08.430381+0000","last_undegraded":"2024-07-26T06:15:08.430381+0000","last_fullsized":"2024-07-26T06:15:08.430381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:44.629861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.443041+0000","last_change":"2024-07-26T06:15:06.413462+0000","last_active":"2024-07-26T06:15:08.443041+0000","last_peered":"2024-07-26T06:15:08.443041+0000","last_clean":"2024-07-26T06:15:08.443041+0000","last_became_active":"2024-07-26T06:15:06.413050+0000","last_became_peered":"2024-07-26T06:15:06.413050+0000","last_unstale":"2024-07-26T06:15:08.443041+0000","last_undegraded":"2024-07-26T06:15:08.443041+0000","last_fullsized":"2024-07-26T06:15:08.443041+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:55:26.398849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063616+0000","last_change":"2024-07-26T06:15:07.452021+0000","last_active":"2024-07-26T06:15:09.063616+0000","last_peered":"2024-07-26T06:15:09.063616+0000","last_clean":"2024-07-26T06:15:09.063616+0000","last_became_active":"2024-07-26T06:15:07.451697+0000","last_became_peered":"2024-07-26T06:15:07.451697+0000","last_unstale":"2024-07-26T06:15:09.063616+0000","last_undegraded":"2024-07-26T06:15:09.063616+0000","last_fullsized":"2024-07-26T06:15:09.063616+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:50.773867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165583+0000","last_change":"2024-07-26T06:15:06.434157+0000","last_active":"2024-07-26T06:15:10.165583+0000","last_peered":"2024-07-26T06:15:10.165583+0000","last_clean":"2024-07-26T06:15:10.165583+0000","last_became_active":"2024-07-26T06:15:06.433764+0000","last_became_peered":"2024-07-26T06:15:06.433764+0000","last_unstale":"2024-07-26T06:15:10.165583+0000","last_undegraded":"2024-07-26T06:15:10.165583+0000","last_fullsized":"2024-07-26T06:15:10.165583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:55.743833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063214+0000","last_change":"2024-07-26T06:15:07.453298+0000","last_active":"2024-07-26T06:15:09.063214+0000","last_peered":"2024-07-26T06:15:09.063214+0000","last_clean":"2024-07-26T06:15:09.063214+0000","last_became_active":"2024-07-26T06:15:07.452296+0000","last_became_peered":"2024-07-26T06:15:07.452296+0000","last_unstale":"2024-07-26T06:15:09.063214+0000","last_undegraded":"2024-07-26T06:15:09.063214+0000","last_fullsized":"2024-07-26T06:15:09.063214+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:38.857932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432483+0000","last_change":"2024-07-26T06:15:06.431291+0000","last_active":"2024-07-26T06:15:08.432483+0000","last_peered":"2024-07-26T06:15:08.432483+0000","last_clean":"2024-07-26T06:15:08.432483+0000","last_became_active":"2024-07-26T06:15:06.430987+0000","last_became_peered":"2024-07-26T06:15:06.430987+0000","last_unstale":"2024-07-26T06:15:08.432483+0000","last_undegraded":"2024-07-26T06:15:08.432483+0000","last_fullsized":"2024-07-26T06:15:08.432483+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:27.051477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431531+0000","last_change":"2024-07-26T06:15:07.428559+0000","last_active":"2024-07-26T06:15:08.431531+0000","last_peered":"2024-07-26T06:15:08.431531+0000","last_clean":"2024-07-26T06:15:08.431531+0000","last_became_active":"2024-07-26T06:15:07.427951+0000","last_became_peered":"2024-07-26T06:15:07.427951+0000","last_unstale":"2024-07-26T06:15:08.431531+0000","last_undegraded":"2024-07-26T06:15:08.431531+0000","last_fullsized":"2024-07-26T06:15:08.431531+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:41.845575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.433070+0000","last_change":"2024-07-26T06:15:06.431511+0000","last_active":"2024-07-26T06:15:08.433070+0000","last_peered":"2024-07-26T06:15:08.433070+0000","last_clean":"2024-07-26T06:15:08.433070+0000","last_became_active":"2024-07-26T06:15:06.430043+0000","last_became_peered":"2024-07-26T06:15:06.430043+0000","last_unstale":"2024-07-26T06:15:08.433070+0000","last_undegraded":"2024-07-26T06:15:08.433070+0000","last_fullsized":"2024-07-26T06:15:08.433070+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:21:25.074600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162693+0000","last_change":"2024-07-26T06:15:07.431238+0000","last_active":"2024-07-26T06:15:10.162693+0000","last_peered":"2024-07-26T06:15:10.162693+0000","last_clean":"2024-07-26T06:15:10.162693+0000","last_became_active":"2024-07-26T06:15:07.430655+0000","last_became_peered":"2024-07-26T06:15:07.430655+0000","last_unstale":"2024-07-26T06:15:10.162693+0000","last_undegraded":"2024-07-26T06:15:10.162693+0000","last_fullsized":"2024-07-26T06:15:10.162693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:39:41.396598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.164887+0000","last_change":"2024-07-26T06:15:06.432150+0000","last_active":"2024-07-26T06:15:10.164887+0000","last_peered":"2024-07-26T06:15:10.164887+0000","last_clean":"2024-07-26T06:15:10.164887+0000","last_became_active":"2024-07-26T06:15:06.431043+0000","last_became_peered":"2024-07-26T06:15:06.431043+0000","last_unstale":"2024-07-26T06:15:10.164887+0000","last_undegraded":"2024-07-26T06:15:10.164887+0000","last_fullsized":"2024-07-26T06:15:10.164887+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:59:40.569992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427922+0000","last_change":"2024-07-26T06:15:07.445135+0000","last_active":"2024-07-26T06:15:08.427922+0000","last_peered":"2024-07-26T06:15:08.427922+0000","last_clean":"2024-07-26T06:15:08.427922+0000","last_became_active":"2024-07-26T06:15:07.444835+0000","last_became_peered":"2024-07-26T06:15:07.444835+0000","last_unstale":"2024-07-26T06:15:08.427922+0000","last_undegraded":"2024-07-26T06:15:08.427922+0000","last_fullsized":"2024-07-26T06:15:08.427922+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:27.950292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427439+0000","last_change":"2024-07-26T06:15:06.428563+0000","last_active":"2024-07-26T06:15:08.427439+0000","last_peered":"2024-07-26T06:15:08.427439+0000","last_clean":"2024-07-26T06:15:08.427439+0000","last_became_active":"2024-07-26T06:15:06.427905+0000","last_became_peered":"2024-07-26T06:15:06.427905+0000","last_unstale":"2024-07-26T06:15:08.427439+0000","last_undegraded":"2024-07-26T06:15:08.427439+0000","last_fullsized":"2024-07-26T06:15:08.427439+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:07:43.751746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427140+0000","last_change":"2024-07-26T06:15:07.427333+0000","last_active":"2024-07-26T06:15:08.427140+0000","last_peered":"2024-07-26T06:15:08.427140+0000","last_clean":"2024-07-26T06:15:08.427140+0000","last_became_active":"2024-07-26T06:15:07.426998+0000","last_became_peered":"2024-07-26T06:15:07.426998+0000","last_unstale":"2024-07-26T06:15:08.427140+0000","last_undegraded":"2024-07-26T06:15:08.427140+0000","last_fullsized":"2024-07-26T06:15:08.427140+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:53:26.197289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.427912+0000","last_change":"2024-07-26T06:15:06.429477+0000","last_active":"2024-07-26T06:15:08.427912+0000","last_peered":"2024-07-26T06:15:08.427912+0000","last_clean":"2024-07-26T06:15:08.427912+0000","last_became_active":"2024-07-26T06:15:06.429001+0000","last_became_peered":"2024-07-26T06:15:06.429001+0000","last_unstale":"2024-07-26T06:15:08.427912+0000","last_undegraded":"2024-07-26T06:15:08.427912+0000","last_fullsized":"2024-07-26T06:15:08.427912+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:50.337732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:09.063856+0000","last_change":"2024-07-26T06:15:07.447128+0000","last_active":"2024-07-26T06:15:09.063856+0000","last_peered":"2024-07-26T06:15:09.063856+0000","last_clean":"2024-07-26T06:15:09.063856+0000","last_became_active":"2024-07-26T06:15:07.446757+0000","last_became_peered":"2024-07-26T06:15:07.446757+0000","last_unstale":"2024-07-26T06:15:09.063856+0000","last_undegraded":"2024-07-26T06:15:09.063856+0000","last_fullsized":"2024-07-26T06:15:09.063856+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:16:49.612701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432647+0000","last_change":"2024-07-26T06:15:06.423800+0000","last_active":"2024-07-26T06:15:08.432647+0000","last_peered":"2024-07-26T06:15:08.432647+0000","last_clean":"2024-07-26T06:15:08.432647+0000","last_became_active":"2024-07-26T06:15:06.423480+0000","last_became_peered":"2024-07-26T06:15:06.423480+0000","last_unstale":"2024-07-26T06:15:08.432647+0000","last_undegraded":"2024-07-26T06:15:08.432647+0000","last_fullsized":"2024-07-26T06:15:08.432647+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:24.551356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428971+0000","last_change":"2024-07-26T06:15:07.415883+0000","last_active":"2024-07-26T06:15:08.428971+0000","last_peered":"2024-07-26T06:15:08.428971+0000","last_clean":"2024-07-26T06:15:08.428971+0000","last_became_active":"2024-07-26T06:15:07.415464+0000","last_became_peered":"2024-07-26T06:15:07.415464+0000","last_unstale":"2024-07-26T06:15:08.428971+0000","last_undegraded":"2024-07-26T06:15:08.428971+0000","last_fullsized":"2024-07-26T06:15:08.428971+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:58.101987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429649+0000","last_change":"2024-07-26T06:15:06.440175+0000","last_active":"2024-07-26T06:15:08.429649+0000","last_peered":"2024-07-26T06:15:08.429649+0000","last_clean":"2024-07-26T06:15:08.429649+0000","last_became_active":"2024-07-26T06:15:06.439829+0000","last_became_peered":"2024-07-26T06:15:06.439829+0000","last_unstale":"2024-07-26T06:15:08.429649+0000","last_undegraded":"2024-07-26T06:15:08.429649+0000","last_fullsized":"2024-07-26T06:15:08.429649+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:57.096105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.429434+0000","last_change":"2024-07-26T06:15:07.437465+0000","last_active":"2024-07-26T06:15:08.429434+0000","last_peered":"2024-07-26T06:15:08.429434+0000","last_clean":"2024-07-26T06:15:08.429434+0000","last_became_active":"2024-07-26T06:15:07.437100+0000","last_became_peered":"2024-07-26T06:15:07.437100+0000","last_unstale":"2024-07-26T06:15:08.429434+0000","last_undegraded":"2024-07-26T06:15:08.429434+0000","last_fullsized":"2024-07-26T06:15:08.429434+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:28:36.803795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162482+0000","last_change":"2024-07-26T06:15:06.428572+0000","last_active":"2024-07-26T06:15:10.162482+0000","last_peered":"2024-07-26T06:15:10.162482+0000","last_clean":"2024-07-26T06:15:10.162482+0000","last_became_active":"2024-07-26T06:15:06.428279+0000","last_became_peered":"2024-07-26T06:15:06.428279+0000","last_unstale":"2024-07-26T06:15:10.162482+0000","last_undegraded":"2024-07-26T06:15:10.162482+0000","last_fullsized":"2024-07-26T06:15:10.162482+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:50:49.781379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.431623+0000","last_change":"2024-07-26T06:15:07.428752+0000","last_active":"2024-07-26T06:15:08.431623+0000","last_peered":"2024-07-26T06:15:08.431623+0000","last_clean":"2024-07-26T06:15:08.431623+0000","last_became_active":"2024-07-26T06:15:07.428327+0000","last_became_peered":"2024-07-26T06:15:07.428327+0000","last_unstale":"2024-07-26T06:15:08.431623+0000","last_undegraded":"2024-07-26T06:15:08.431623+0000","last_fullsized":"2024-07-26T06:15:08.431623+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:07.323166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428956+0000","last_change":"2024-07-26T06:15:06.434053+0000","last_active":"2024-07-26T06:15:08.428956+0000","last_peered":"2024-07-26T06:15:08.428956+0000","last_clean":"2024-07-26T06:15:08.428956+0000","last_became_active":"2024-07-26T06:15:06.433743+0000","last_became_peered":"2024-07-26T06:15:06.433743+0000","last_unstale":"2024-07-26T06:15:08.428956+0000","last_undegraded":"2024-07-26T06:15:08.428956+0000","last_fullsized":"2024-07-26T06:15:08.428956+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:55.743833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.432303+0000","last_change":"2024-07-26T06:15:07.434188+0000","last_active":"2024-07-26T06:15:08.432303+0000","last_peered":"2024-07-26T06:15:08.432303+0000","last_clean":"2024-07-26T06:15:08.432303+0000","last_became_active":"2024-07-26T06:15:07.433895+0000","last_became_peered":"2024-07-26T06:15:07.433895+0000","last_unstale":"2024-07-26T06:15:08.432303+0000","last_undegraded":"2024-07-26T06:15:08.432303+0000","last_fullsized":"2024-07-26T06:15:08.432303+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:08.498466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442622+0000","last_change":"2024-07-26T06:15:06.419916+0000","last_active":"2024-07-26T06:15:08.442622+0000","last_peered":"2024-07-26T06:15:08.442622+0000","last_clean":"2024-07-26T06:15:08.442622+0000","last_became_active":"2024-07-26T06:15:06.419614+0000","last_became_peered":"2024-07-26T06:15:06.419614+0000","last_unstale":"2024-07-26T06:15:08.442622+0000","last_undegraded":"2024-07-26T06:15:08.442622+0000","last_fullsized":"2024-07-26T06:15:08.442622+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:50:49.781379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.165051+0000","last_change":"2024-07-26T06:15:07.441831+0000","last_active":"2024-07-26T06:15:10.165051+0000","last_peered":"2024-07-26T06:15:10.165051+0000","last_clean":"2024-07-26T06:15:10.165051+0000","last_became_active":"2024-07-26T06:15:07.440883+0000","last_became_peered":"2024-07-26T06:15:07.440883+0000","last_unstale":"2024-07-26T06:15:10.165051+0000","last_undegraded":"2024-07-26T06:15:10.165051+0000","last_fullsized":"2024-07-26T06:15:10.165051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:07:02.236299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.162501+0000","last_change":"2024-07-26T06:15:06.424010+0000","last_active":"2024-07-26T06:15:10.162501+0000","last_peered":"2024-07-26T06:15:10.162501+0000","last_clean":"2024-07-26T06:15:10.162501+0000","last_became_active":"2024-07-26T06:15:06.423692+0000","last_became_peered":"2024-07-26T06:15:06.423692+0000","last_unstale":"2024-07-26T06:15:10.162501+0000","last_undegraded":"2024-07-26T06:15:10.162501+0000","last_fullsized":"2024-07-26T06:15:10.162501+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:03:46.856382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.442216+0000","last_change":"2024-07-26T06:15:07.416039+0000","last_active":"2024-07-26T06:15:08.442216+0000","last_peered":"2024-07-26T06:15:08.442216+0000","last_clean":"2024-07-26T06:15:08.442216+0000","last_became_active":"2024-07-26T06:15:07.414833+0000","last_became_peered":"2024-07-26T06:15:07.414833+0000","last_unstale":"2024-07-26T06:15:08.442216+0000","last_undegraded":"2024-07-26T06:15:08.442216+0000","last_fullsized":"2024-07-26T06:15:08.442216+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:37:22.010597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428007+0000","last_change":"2024-07-26T06:15:06.425341+0000","last_active":"2024-07-26T06:15:08.428007+0000","last_peered":"2024-07-26T06:15:08.428007+0000","last_clean":"2024-07-26T06:15:08.428007+0000","last_became_active":"2024-07-26T06:15:06.424843+0000","last_became_peered":"2024-07-26T06:15:06.424843+0000","last_unstale":"2024-07-26T06:15:08.428007+0000","last_undegraded":"2024-07-26T06:15:08.428007+0000","last_fullsized":"2024-07-26T06:15:08.428007+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:26.944410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.430123+0000","last_change":"2024-07-26T06:15:07.441572+0000","last_active":"2024-07-26T06:15:08.430123+0000","last_peered":"2024-07-26T06:15:08.430123+0000","last_clean":"2024-07-26T06:15:08.430123+0000","last_became_active":"2024-07-26T06:15:07.440083+0000","last_became_peered":"2024-07-26T06:15:07.440083+0000","last_unstale":"2024-07-26T06:15:08.430123+0000","last_undegraded":"2024-07-26T06:15:08.430123+0000","last_fullsized":"2024-07-26T06:15:08.430123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:06.377499+0000","last_clean_scrub_stamp":"2024-07-26T06:15:06.377499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:07:02.236299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:10.166175+0000","last_change":"2024-07-26T06:15:06.428716+0000","last_active":"2024-07-26T06:15:10.166175+0000","last_peered":"2024-07-26T06:15:10.166175+0000","last_clean":"2024-07-26T06:15:10.166175+0000","last_became_active":"2024-07-26T06:15:06.427613+0000","last_became_peered":"2024-07-26T06:15:06.427613+0000","last_unstale":"2024-07-26T06:15:10.166175+0000","last_undegraded":"2024-07-26T06:15:10.166175+0000","last_fullsized":"2024-07-26T06:15:10.166175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:15:05.371617+0000","last_clean_scrub_stamp":"2024-07-26T06:15:05.371617+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:53:25.191407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":153,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-07-26T06:15:08.428127+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:15:08.428127+0000","last_peered":"2024-07-26T06:15:08.428127+0000","last_clean":"2024-07-26T06:15:08.428127+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:15:08.428127+0000","last_undegraded":"2024-07-26T06:15:08.428127+0000","last_fullsized":"2024-07-26T06:15:08.428127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":801163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":211377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":210775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":536576,"data_stored":212339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509466,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":801107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542169,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":212317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":211343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":210775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:15:14.360 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:15:14.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.610+0000 7f3740c00640 1 -- 172.21.15.163:0/927184145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 msgr2=0x7f373c0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.610+0000 7f3740c00640 1 --2- 172.21.15.163:0/927184145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c0b3710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f373c06ce80 tx=0x7f373402aac0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.611+0000 7f3740c00640 1 -- 172.21.15.163:0/927184145 shutdown_connections 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.611+0000 7f3740c00640 1 --2- 172.21.15.163:0/927184145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c0b3710 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.611+0000 7f3740c00640 1 --2- 172.21.15.163:0/927184145 >> v2:172.21.15.164:3300/0 conn(0x7f373c1556f0 0x7f373c0cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.611+0000 7f3740c00640 1 -- 172.21.15.163:0/927184145 >> 172.21.15.163:0/927184145 conn(0x7f373c0b0e30 msgr2=0x7f373c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.611+0000 7f3740c00640 1 -- 172.21.15.163:0/927184145 shutdown_connections 2024-07-26T06:15:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.611+0000 7f3740c00640 1 -- 172.21.15.163:0/927184145 wait complete. 2024-07-26T06:15:14.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.613+0000 7f3740c00640 1 Processor -- start 2024-07-26T06:15:14.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.613+0000 7f3740c00640 1 -- start start 2024-07-26T06:15:14.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.614+0000 7f3740c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c1ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.614+0000 7f3740c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f373c0ce2d0 0x7f373c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.614+0000 7f3740c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f373c1556f0 0x7f373c0ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.614+0000 7f3740c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f373c0b3fe0 con 0x7f373c071c40 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.614+0000 7f3740c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f373c0b3e60 con 0x7f373c0ce2d0 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.614+0000 7f3740c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f373c0b4160 con 0x7f373c1556f0 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c1ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c1ee350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55458/0 (socket says 172.21.15.163:55458) 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 -- 172.21.15.163:0/2374345463 learned_addr learned my addr 172.21.15.163:0/2374345463 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373aa00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f373c0ce2d0 0x7f373c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 -- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f373c1556f0 msgr2=0x7f373c0ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f373c1556f0 0x7f373c0ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 -- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f373c0ce2d0 msgr2=0x7f373c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f373c0ce2d0 0x7f373c0ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 -- 172.21.15.163:0/2374345463 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c1f41e0 con 0x7f373c071c40 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373aa00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f373c0ce2d0 0x7f373c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.615+0000 7f373b400640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c1ee350 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d770 tx=0x7f372c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.616+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c017070 con 0x7f373c071c40 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.616+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f372c009d20 con 0x7f373c071c40 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.616+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c0106c0 con 0x7f373c071c40 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.616+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c153060 con 0x7f373c071c40 2024-07-26T06:15:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.616+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c1f4780 con 0x7f373c071c40 2024-07-26T06:15:14.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.617+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373c0b46f0 con 0x7f373c071c40 2024-07-26T06:15:14.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.618+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f372c01a070 con 0x7f373c071c40 2024-07-26T06:15:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.619+0000 7f3733400640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3718076480 0x7f3718078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.619+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f372c097ca0 con 0x7f373c071c40 2024-07-26T06:15:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.620+0000 7f373aa00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3718076480 0x7f3718078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:14.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.621+0000 7f373aa00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3718076480 0x7f3718078940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f373c15df10 tx=0x7f3724009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:14.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.623+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f372c061220 con 0x7f373c071c40 2024-07-26T06:15:14.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.903+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f373c0b34f0 con 0x7f373c071c40 2024-07-26T06:15:14.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.907+0000 7f3733400640 1 -- 172.21.15.163:0/2374345463 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f372c060b20 con 0x7f373c071c40 2024-07-26T06:15:14.970 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:15:14.970 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:15:14.970 INFO:teuthology.orchestra.run.smithi163.stdout: caps mds = "allow" 2024-07-26T06:15:14.970 INFO:teuthology.orchestra.run.smithi163.stdout: caps mon = "allow r" 2024-07-26T06:15:14.970 INFO:teuthology.orchestra.run.smithi163.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:updated caps for client.0 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3718076480 msgr2=0x7f3718078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3718076480 0x7f3718078940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f373c15df10 tx=0x7f3724009040 comp rx=0 tx=0).stop 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 msgr2=0x7f373c1ee350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c1ee350 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d770 tx=0x7f372c00dc30 comp rx=0 tx=0).stop 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 shutdown_connections 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3718076480 0x7f3718078940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f373c1556f0 0x7f373c0ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f373c0ce2d0 0x7f373c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 --2- 172.21.15.163:0/2374345463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f373c071c40 0x7f373c1ee350 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:14.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 >> 172.21.15.163:0/2374345463 conn(0x7f373c0b0e30 msgr2=0x7f373c15df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:14.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 shutdown_connections 2024-07-26T06:15:14.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:14.910+0000 7f3740c00640 1 -- 172.21.15.163:0/2374345463 wait complete. 2024-07-26T06:15:14.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13d1c00640 1 -- 172.21.15.163:0/3168797390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 msgr2=0x7f13b0005910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3168797390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13b0005910 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009b50 tx=0x7f13b0006940 comp rx=0 tx=0).stop 2024-07-26T06:15:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13cb400640 1 -- 172.21.15.163:0/3168797390 reap_dead start 2024-07-26T06:15:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13d1c00640 1 -- 172.21.15.163:0/3168797390 shutdown_connections 2024-07-26T06:15:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13d1c00640 1 -- 172.21.15.163:0/3168797390 >> 172.21.15.163:0/3168797390 conn(0x7f13cc06d420 msgr2=0x7f13cc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13d1c00640 1 -- 172.21.15.163:0/3168797390 shutdown_connections 2024-07-26T06:15:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.178+0000 7f13d1c00640 1 -- 172.21.15.163:0/3168797390 wait complete. 2024-07-26T06:15:15.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.180+0000 7f13d1c00640 1 Processor -- start 2024-07-26T06:15:15.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.180+0000 7f13d1c00640 1 -- start start 2024-07-26T06:15:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13d1c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13cc1f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13d1c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc0adb50 0x7f13cc0c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13d1c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc0adfd0 0x7f13cc152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13d1c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc0cd7f0 con 0x7f13b0005160 2024-07-26T06:15:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13d1c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc0cd670 con 0x7f13cc0adb50 2024-07-26T06:15:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13d1c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f13cc0cd970 con 0x7f13cc0adfd0 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13caa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13cc1f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13caa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13cc1f3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53340/0 (socket says 172.21.15.163:53340) 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.181+0000 7f13caa00640 1 -- 172.21.15.163:0/3382102923 learned_addr learned my addr 172.21.15.163:0/3382102923 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13caa00640 1 -- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc0adfd0 msgr2=0x7f13cc152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13caa00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc0adfd0 0x7f13cc152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13caa00640 1 -- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc0adb50 msgr2=0x7f13cc0c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13caa00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc0adb50 0x7f13cc0c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13caa00640 1 -- 172.21.15.163:0/3382102923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13cc1559b0 con 0x7f13b0005160 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13caa00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13cc1f3a50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00d8f0 tx=0x7f13bc00ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc016020 con 0x7f13b0005160 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13bc00b8d0 con 0x7f13b0005160 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13cc072340 con 0x7f13b0005160 2024-07-26T06:15:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.182+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc00bd20 con 0x7f13b0005160 2024-07-26T06:15:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.183+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13cc072900 con 0x7f13b0005160 2024-07-26T06:15:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.184+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13cc0b40a0 con 0x7f13b0005160 2024-07-26T06:15:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.185+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f13bc019020 con 0x7f13b0005160 2024-07-26T06:15:15.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.186+0000 7f13c2a00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13a80764b0 0x7f13a8078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:15.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.186+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f13bc097c20 con 0x7f13b0005160 2024-07-26T06:15:15.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.186+0000 7f13ca000640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13a80764b0 0x7f13a8078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:15.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.187+0000 7f13ca000640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13a80764b0 0x7f13a8078970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f13b4035000 tx=0x7f13b4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:15.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.190+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13bc020020 con 0x7f13b0005160 2024-07-26T06:15:15.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.417+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13cc069050 con 0x7f13b0005160 2024-07-26T06:15:15.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.418+0000 7f13c2a00640 1 -- 172.21.15.163:0/3382102923 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f13bc0612b0 con 0x7f13b0005160 2024-07-26T06:15:15.419 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:15.419 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":25,"btime":"2024-07-26T06:15:12:424563+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:15.420 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 25 2024-07-26T06:15:15.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13a80764b0 msgr2=0x7f13a8078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:15.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13a80764b0 0x7f13a8078970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f13b4035000 tx=0x7f13b4004730 comp rx=0 tx=0).stop 2024-07-26T06:15:15.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 msgr2=0x7f13cc1f3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:15.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13cc1f3a50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00d8f0 tx=0x7f13bc00ddb0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 shutdown_connections 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f13a80764b0 0x7f13a8078970 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13cc0adfd0 0x7f13cc152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13cc0adb50 0x7f13cc0c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 --2- 172.21.15.163:0/3382102923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13b0005160 0x7f13cc1f3a50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 >> 172.21.15.163:0/3382102923 conn(0x7f13cc06d420 msgr2=0x7f13cc06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.421+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 shutdown_connections 2024-07-26T06:15:15.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:15.422+0000 7f13d1c00640 1 -- 172.21.15.163:0/3382102923 wait complete. 2024-07-26T06:15:15.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:15:06.868107+0000', 'modified': '2024-07-26T06:15:11.906262+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5256': {'gid': 5256, 'name': 'b', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6835/1618066632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 1618066632}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 1618066632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5256, 'qdb_cluster': [5256]} 2024-07-26T06:15:15.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:15:15.848 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 -- 172.21.15.163:0/4004334108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f079415d5a0 msgr2=0x7f07940ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 --2- 172.21.15.163:0/4004334108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f079415d5a0 0x7f07940ce2f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0794160e50 tx=0x7f078802a4d0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 -- 172.21.15.163:0/4004334108 shutdown_connections 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 --2- 172.21.15.163:0/4004334108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07940b1e80 0x7f07940b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 --2- 172.21.15.163:0/4004334108 >> v2:172.21.15.164:3300/0 conn(0x7f07940ce830 0x7f07940b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 --2- 172.21.15.163:0/4004334108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f079415d5a0 0x7f07940ce2f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.060+0000 7f079a800640 1 -- 172.21.15.163:0/4004334108 >> 172.21.15.163:0/4004334108 conn(0x7f07940cfda0 msgr2=0x7f07940d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.061+0000 7f079a800640 1 -- 172.21.15.163:0/4004334108 shutdown_connections 2024-07-26T06:15:16.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.061+0000 7f079a800640 1 -- 172.21.15.163:0/4004334108 wait complete. 2024-07-26T06:15:16.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.062+0000 7f079a800640 1 Processor -- start 2024-07-26T06:15:16.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.062+0000 7f079a800640 1 -- start start 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.063+0000 7f079a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 0x7f07941f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.063+0000 7f079a800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07940ce830 0x7f07940b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.063+0000 7f079a800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f079415d5a0 0x7f07940b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.063+0000 7f079a800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07940b2760 con 0x7f07940b1e80 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.063+0000 7f079a800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07940b25e0 con 0x7f079415d5a0 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.063+0000 7f079a800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f07940b4da0 con 0x7f07940ce830 2024-07-26T06:15:16.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 0x7f07941f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 0x7f07941f32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53370/0 (socket says 172.21.15.163:53370) 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 -- 172.21.15.163:0/1112012460 learned_addr learned my addr 172.21.15.163:0/1112012460 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0798a00640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07940ce830 0x7f07940b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799e00640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f079415d5a0 0x7f07940b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 -- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07940ce830 msgr2=0x7f07940b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07940ce830 0x7f07940b8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 -- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f079415d5a0 msgr2=0x7f07940b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f079415d5a0 0x7f07940b86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.064+0000 7f0799400640 1 -- 172.21.15.163:0/1112012460 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07940cbdb0 con 0x7f07940b1e80 2024-07-26T06:15:16.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.065+0000 7f0799400640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 0x7f07941f32f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0788034040 tx=0x7f07880039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.065+0000 7f0791600640 1 -- 172.21.15.163:0/1112012460 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078803e030 con 0x7f07940b1e80 2024-07-26T06:15:16.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.065+0000 7f0791600640 1 -- 172.21.15.163:0/1112012460 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07880047a0 con 0x7f07940b1e80 2024-07-26T06:15:16.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.065+0000 7f0791600640 1 -- 172.21.15.163:0/1112012460 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788035040 con 0x7f07940b1e80 2024-07-26T06:15:16.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.065+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07940cb070 con 0x7f07940b1e80 2024-07-26T06:15:16.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.065+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07941f3830 con 0x7f07940b1e80 2024-07-26T06:15:16.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.067+0000 7f0791600640 1 -- 172.21.15.163:0/1112012460 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f0788041030 con 0x7f07940b1e80 2024-07-26T06:15:16.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.068+0000 7f0791600640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0778076480 0x7f0778078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.069+0000 7f0791600640 1 -- 172.21.15.163:0/1112012460 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f07880b8380 con 0x7f07940b1e80 2024-07-26T06:15:16.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.069+0000 7f079ec28740 1 Processor -- start 2024-07-26T06:15:16.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.069+0000 7f079ec28740 1 -- start start 2024-07-26T06:15:16.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.069+0000 7f0798a00640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0778076480 0x7f0778078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d9201940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f079ec28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 0x5602d91feea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91bbe50 0x5602d91b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f079ec28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602d90d13f0 con 0x5602d92008d0 2024-07-26T06:15:16.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5602d912c5c0 con 0x5602d9201560 2024-07-26T06:15:16.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5602d911bb60 con 0x5602d91bbe50 2024-07-26T06:15:16.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f0798a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 0x5602d91feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f0798a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 0x5602d91feea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53378/0 (socket says 172.21.15.163:53378) 2024-07-26T06:15:16.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.070+0000 7f0798a00640 1 -- 172.21.15.163:0/1710950866 learned_addr learned my addr 172.21.15.163:0/1710950866 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:16.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0778076480 0x7f0778078940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0784007350 tx=0x7f07840072e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 -- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91bbe50 msgr2=0x5602d91b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0799e00640 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91bbe50 0x5602d91b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91bbe50 0x5602d91b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 -- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 msgr2=0x5602d9201940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d9201940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 -- 172.21.15.163:0/1710950866 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d8e80850 con 0x5602d92008d0 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0798a00640 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 0x5602d91feea0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0784010590 tx=0x7f0784011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b91ad15e3e63da server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.071+0000 7f0781c00640 1 -- 172.21.15.163:0/1710950866 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784021070 con 0x5602d92008d0 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f0781c00640 1 -- 172.21.15.163:0/1710950866 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f078400fa80 con 0x5602d92008d0 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f0781c00640 1 -- 172.21.15.163:0/1710950866 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078401d450 con 0x5602d92008d0 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 -- 172.21.15.163:0/1710950866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 msgr2=0x5602d91feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 0x5602d91feea0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0784010590 tx=0x7f0784011000 comp rx=0 tx=0).stop 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 -- 172.21.15.163:0/1710950866 shutdown_connections 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91bbe50 0x5602d91b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d92008d0 0x5602d91feea0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 --2- 172.21.15.163:0/1710950866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d9201940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 -- 172.21.15.163:0/1710950866 >> 172.21.15.163:0/1710950866 conn(0x5602d91b71a0 msgr2=0x5602d91ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 -- 172.21.15.163:0/1710950866 shutdown_connections 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.072+0000 7f079ec28740 1 -- 172.21.15.163:0/1710950866 wait complete. 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.073+0000 7f079ec28740 1 -- start start 2024-07-26T06:15:16.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.073+0000 7f079ec28740 1 Processor -- start 2024-07-26T06:15:16.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.074+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d9201560 0x5602d91b8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.074+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91db420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.074+0000 7f079ec28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91e0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.074+0000 7f079ec28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602d90c4210 con 0x5602d91e0790 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.074+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5602d911fba0 con 0x5602d91e2620 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.074+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5602d91ffce0 con 0x5602d9201560 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f079ec28740 10 client.0 _ll_register_callbacks cb 0x5602d90c48a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91e0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91e0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53392/0 (socket says 172.21.15.163:53392) 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 -- 172.21.15.163:0/1629346208 learned_addr learned my addr 172.21.15.163:0/1629346208 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799400640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d9201560 0x5602d91b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0798a00640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91db420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 -- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d9201560 msgr2=0x5602d91b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d9201560 0x5602d91b8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 -- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 msgr2=0x5602d91db420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91db420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 -- 172.21.15.163:0/1629346208 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d8ef76b0 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799400640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d9201560 0x5602d91b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0799e00640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91e0c10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f078c00bf90 tx=0x7f078c00d650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4925c443788d1c7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.075+0000 7f0776000640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f0776000640 1 -- 172.21.15.163:0/1629346208 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c016070 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f0776000640 1 -- 172.21.15.163:0/1629346208 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f078c012870 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5602d912e510 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f0776000640 1 -- 172.21.15.163:0/1629346208 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f078c021050 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 10 client.5382 fetch_fsmap learned FSMap version 25 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5602d91b9a00 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f0776000640 1 -- 172.21.15.163:0/1629346208 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x5602d912e510 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f0776000640 1 -- 172.21.15.163:0/1629346208 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f078c024020 con 0x5602d91e0790 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 10 client.5382 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 10 client.5382 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:15:16.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 20 client.5382 populate_metadata read hostname 'smithi163' 2024-07-26T06:15:16.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.076+0000 7f079ec28740 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9213260 0x5602d91e47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.077+0000 7f079ec28740 4 client.5382 mds_command: new command op to 5256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:15:16.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.077+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 --> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602d9129fb0 con 0x5602d9213260 2024-07-26T06:15:16.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.077+0000 7f0799400640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9213260 0x5602d91e47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.078+0000 7f0799400640 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9213260 0x5602d91e47b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fddb4fb89b348d02 server_cookie=aed5f774077ca40f in_seq=0 out_seq=0 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.078+0000 7f0776000640 10 client.5382 ms_handle_connect on v2:172.21.15.164:6834/1618066632 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f0776000640 1 -- 172.21.15.163:0/1629346208 <== mds.0 v2:172.21.15.164:6834/1618066632 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5602d9129fb0 con 0x5602d9213260 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f0776000640 10 client.5382 handle_command_reply: tid=0 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 client.5382 shutdown 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 msgr2=0x5602d91e0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91e0c10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f078c00bf90 tx=0x7f078c00d650 comp rx=0 tx=0).stop 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 shutdown_connections 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91e0c10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91db420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9213260 0x5602d91e47b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 --2- 172.21.15.163:0/1629346208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d9201560 0x5602d91b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 >> 172.21.15.163:0/1629346208 conn(0x5602d91bbe50 msgr2=0x5602d9200170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f0776000640 0 client.5382 ms_handle_reset on v2:172.21.15.164:6834/1618066632 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 shutdown_connections 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 1 -- 172.21.15.163:0/1629346208 wait complete. 2024-07-26T06:15:16.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.081+0000 7f079ec28740 20 client.5382 trim_cache size 0 max 16384 2024-07-26T06:15:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.102+0000 7f079ec28740 1 Processor -- start 2024-07-26T06:15:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.102+0000 7f079ec28740 1 -- start start 2024-07-26T06:15:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e0790 0x5602d91b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91b8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f079ec28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 0x5602d9221ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f079ec28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07941ff240 con 0x5602d9201560 2024-07-26T06:15:16.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07941fe5d0 con 0x5602d91e2620 2024-07-26T06:15:16.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f07941fec20 con 0x5602d91e0790 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f0799e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 0x5602d9221ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f0799e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 0x5602d9221ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53394/0 (socket says 172.21.15.163:53394) 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.103+0000 7f0799e00640 1 -- 172.21.15.163:0/4047193028 learned_addr learned my addr 172.21.15.163:0/4047193028 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0798a00640 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799400640 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e0790 0x5602d91b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799e00640 1 -- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e0790 msgr2=0x5602d91b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799e00640 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e0790 0x5602d91b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799e00640 1 -- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 msgr2=0x5602d91b8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799e00640 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91b8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799e00640 1 -- 172.21.15.163:0/4047193028 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d9212f00 con 0x5602d9201560 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0799e00640 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 0x5602d9221ec0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b190 tx=0x7f078c012670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5aece0a97fa49cc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0781200640 1 -- 172.21.15.163:0/4047193028 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c029030 con 0x5602d9201560 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0781200640 1 -- 172.21.15.163:0/4047193028 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f078c012e50 con 0x5602d9201560 2024-07-26T06:15:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.104+0000 7f0781200640 1 -- 172.21.15.163:0/4047193028 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c016040 con 0x5602d9201560 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 -- 172.21.15.163:0/4047193028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 msgr2=0x5602d9221ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 0x5602d9221ec0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b190 tx=0x7f078c012670 comp rx=0 tx=0).stop 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 -- 172.21.15.163:0/4047193028 shutdown_connections 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d9201560 0x5602d9221ec0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d91e2620 0x5602d91b8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 --2- 172.21.15.163:0/4047193028 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e0790 0x5602d91b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 -- 172.21.15.163:0/4047193028 >> 172.21.15.163:0/4047193028 conn(0x5602d91bbe50 msgr2=0x5602d91fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 -- 172.21.15.163:0/4047193028 shutdown_connections 2024-07-26T06:15:16.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.105+0000 7f079ec28740 1 -- 172.21.15.163:0/4047193028 wait complete. 2024-07-26T06:15:16.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.106+0000 7f079ec28740 1 -- start start 2024-07-26T06:15:16.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.106+0000 7f079ec28740 1 Processor -- start 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e2620 0x5602d91fe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d92029d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602d92130e0 con 0x5602d91e0790 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5602d91e4ed0 con 0x5602d9201560 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5602d9213730 con 0x5602d91e2620 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f079ec28740 10 client.0 _ll_register_callbacks cb 0x7f078c00bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f0799400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f0799400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53408/0 (socket says 172.21.15.163:53408) 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.107+0000 7f0799400640 1 -- 172.21.15.163:0/4048029970 learned_addr learned my addr 172.21.15.163:0/4048029970 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:16.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0798a00640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e2620 0x5602d91fe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799e00640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d92029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799400640 1 -- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e2620 msgr2=0x5602d91fe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799400640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e2620 0x5602d91fe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799400640 1 -- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 msgr2=0x5602d92029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799400640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d92029d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799400640 1 -- 172.21.15.163:0/4048029970 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d91b5f10 con 0x5602d91e0790 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0798a00640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e2620 0x5602d91fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799e00640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d92029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0799400640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91b4bd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f078807d850 tx=0x7f07880092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414b1771ccb524e1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0776000640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0776000640 1 -- 172.21.15.163:0/4048029970 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788048040 con 0x5602d91e0790 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f0776000640 1 -- 172.21.15.163:0/4048029970 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f07880b6930 con 0x5602d91e0790 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.108+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f078c012e50 con 0x5602d91e0790 2024-07-26T06:15:16.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f0776000640 1 -- 172.21.15.163:0/4048029970 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f07880b6050 con 0x5602d91e0790 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 10 client.5388 fetch_fsmap learned FSMap version 25 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5602d91b8900 con 0x5602d91e0790 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f0776000640 1 -- 172.21.15.163:0/4048029970 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f078c012e50 con 0x5602d91e0790 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f0776000640 1 -- 172.21.15.163:0/4048029970 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f0788081460 con 0x5602d91e0790 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 10 client.5388 fetch_fsmap finished waiting for FSMap version 25 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 10 client.5388 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 20 client.5388 populate_metadata read hostname 'smithi163' 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9202f10 0x5602d91e2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 4 client.5388 mds_command: new command op to 5256 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.109+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 --> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] -- command(tid 0: {"prefix": "status"}) -- 0x5602d91276f0 con 0x5602d9202f10 2024-07-26T06:15:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.110+0000 7f0798a00640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9202f10 0x5602d91e2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.111+0000 7f0798a00640 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9202f10 0x5602d91e2a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6072f3a8b420270 server_cookie=2086c051db5df0b8 in_seq=0 out_seq=0 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.111+0000 7f0776000640 10 client.5388 ms_handle_connect on v2:172.21.15.164:6834/1618066632 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.111+0000 7f0776000640 1 -- 172.21.15.163:0/4048029970 <== mds.0 v2:172.21.15.164:6834/1618066632 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5602d91276f0 con 0x5602d9202f10 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.111+0000 7f0776000640 10 client.5388 handle_command_reply: tid=0 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.111+0000 7f079ec28740 1 client.5388 shutdown 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 msgr2=0x5602d91b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91b4bd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f078807d850 tx=0x7f07880092d0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 shutdown_connections 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5602d9201560 0x5602d92029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:6834/1618066632,v1:172.21.15.164:6835/1618066632] conn(0x5602d9202f10 0x5602d91e2a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5602d91e2620 0x5602d91fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 --2- 172.21.15.163:0/4048029970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602d91e0790 0x5602d91b4bd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f0776000640 0 client.5388 ms_handle_reset on v2:172.21.15.164:6834/1618066632 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 >> 172.21.15.163:0/4048029970 conn(0x5602d91bbe50 msgr2=0x5602d91b7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 shutdown_connections 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 1 -- 172.21.15.163:0/4048029970 wait complete. 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.112+0000 7f079ec28740 20 client.5388 trim_cache size 0 max 16384 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 5256, 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:15:16.116 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 8.6888316589999999, 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 25, 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 47, 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 45, 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 13.651841812000001 2024-07-26T06:15:16.117 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:15:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0778076480 msgr2=0x7f0778078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0778076480 0x7f0778078940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0784007350 tx=0x7f07840072e0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 msgr2=0x7f07941f32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 0x7f07941f32f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0788034040 tx=0x7f07880039e0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 shutdown_connections 2024-07-26T06:15:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0778076480 0x7f0778078940 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f079415d5a0 0x7f07940b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07940ce830 0x7f07940b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 --2- 172.21.15.163:0/1112012460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07940b1e80 0x7f07941f32f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 >> 172.21.15.163:0/1112012460 conn(0x7f07940cfda0 msgr2=0x7f07940d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 shutdown_connections 2024-07-26T06:15:16.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.113+0000 7f079a800640 1 -- 172.21.15.163:0/1112012460 wait complete. 2024-07-26T06:15:16.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.326+0000 7f893be00640 1 -- 172.21.15.163:0/1879589107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 msgr2=0x7f893c0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.326+0000 7f893be00640 1 --2- 172.21.15.163:0/1879589107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c0b3710 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8934005fd0 tx=0x7f893402aa80 comp rx=0 tx=0).stop 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.327+0000 7f893be00640 1 -- 172.21.15.163:0/1879589107 shutdown_connections 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.327+0000 7f893be00640 1 --2- 172.21.15.163:0/1879589107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c0b3710 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.327+0000 7f893be00640 1 --2- 172.21.15.163:0/1879589107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f893c1556f0 0x7f893c0cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.327+0000 7f893be00640 1 -- 172.21.15.163:0/1879589107 >> 172.21.15.163:0/1879589107 conn(0x7f893c0b0e30 msgr2=0x7f893c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.327+0000 7f893be00640 1 -- 172.21.15.163:0/1879589107 shutdown_connections 2024-07-26T06:15:16.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.327+0000 7f893be00640 1 -- 172.21.15.163:0/1879589107 wait complete. 2024-07-26T06:15:16.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.328+0000 7f893be00640 1 Processor -- start 2024-07-26T06:15:16.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.329+0000 7f893be00640 1 -- start start 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.329+0000 7f893be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c14ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.329+0000 7f893be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0ce2d0 0x7f893c14f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f893c1556f0 0x7f893c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f893c0b4020 con 0x7f893c071c40 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f893c0b3ea0 con 0x7f893c0ce2d0 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f893c0b41a0 con 0x7f893c1556f0 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c14ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c14ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53436/0 (socket says 172.21.15.163:53436) 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 -- 172.21.15.163:0/3886500256 learned_addr learned my addr 172.21.15.163:0/3886500256 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 -- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f893c1556f0 msgr2=0x7f893c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f893c1556f0 0x7f893c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 -- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0ce2d0 msgr2=0x7f893c14f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0ce2d0 0x7f893c14f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 -- 172.21.15.163:0/3886500256 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893c157ff0 con 0x7f893c071c40 2024-07-26T06:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.330+0000 7f893aa00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c14ec80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f892c00d740 tx=0x7f892c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.331+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c017070 con 0x7f893c071c40 2024-07-26T06:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.331+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f892c009d20 con 0x7f893c071c40 2024-07-26T06:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.331+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0106c0 con 0x7f893c071c40 2024-07-26T06:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.331+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893c1582d0 con 0x7f893c071c40 2024-07-26T06:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.331+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893c158890 con 0x7f893c071c40 2024-07-26T06:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.332+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f893c158cd0 con 0x7f893c071c40 2024-07-26T06:15:16.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.333+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f892c01a070 con 0x7f893c071c40 2024-07-26T06:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.335+0000 7f8933400640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8918076480 0x7f8918078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.335+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f892c097cb0 con 0x7f893c071c40 2024-07-26T06:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.335+0000 7f8932000640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8918076480 0x7f8918078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.337+0000 7f8932000640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8918076480 0x7f8918078940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8924000f60 tx=0x7f89240079b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.340+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f892c061230 con 0x7f893c071c40 2024-07-26T06:15:16.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.588+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f893c0b4730 con 0x7f893c071c40 2024-07-26T06:15:16.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.588+0000 7f8933400640 1 -- 172.21.15.163:0/3886500256 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f892c060b30 con 0x7f893c071c40 2024-07-26T06:15:16.589 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8918076480 msgr2=0x7f8918078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8918076480 0x7f8918078940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8924000f60 tx=0x7f89240079b0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 msgr2=0x7f893c14ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c14ec80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f892c00d740 tx=0x7f892c00dc00 comp rx=0 tx=0).stop 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 shutdown_connections 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8918076480 0x7f8918078940 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f893c1556f0 0x7f893c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0ce2d0 0x7f893c14f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 --2- 172.21.15.163:0/3886500256 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f893c071c40 0x7f893c14ec80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:16.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.591+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 >> 172.21.15.163:0/3886500256 conn(0x7f893c0b0e30 msgr2=0x7f893c153bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:16.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.592+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 shutdown_connections 2024-07-26T06:15:16.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16.592+0000 7f893be00640 1 -- 172.21.15.163:0/3886500256 wait complete. 2024-07-26T06:15:16.612 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:15:16.612 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:15:16.612 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:15:16.612 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:15:16.612 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:15:16.612 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:15:16.613 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:15:16.613 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:15:16.613 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:15:16.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:16.613 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:15:16.660 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:15:16.660 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:15:16.661 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:15:16.661 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.661 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:15:16.661 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.661 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42488sec preferred_lft 42488sec 2024-07-26T06:15:16.662 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether c2:4f:6d:14:5d:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:15:16.663 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::c04f:6dff:fe14:5d96/64 scope link 2024-07-26T06:15:16.664 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:16.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:16.664 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:15:16.783 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:15:16.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:16.787 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:15:16.840 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:15:16.840 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:15:16.840 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:15:16.840 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:15:16.840 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:15:16.841 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:15:16.841 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:15:16.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:15:16.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:15:16.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:15:16.841 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:15:16.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:15:17.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:15:17.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:17.050 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:15:17.098 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:15:17.098 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:15:17.099 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:15:17.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:17.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:15:17.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:15:17.171 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:15:17.218 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:15:17.219 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:15:17.219 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:15:17.311 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:15:17.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:17.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:15:17.437 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:15:17.438 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:15:17.438 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:15:17.459 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:15:17.459 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:15:17.459 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:15:17.459 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:15:17.459 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:15:17.489 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:15:17.489 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:15:17.489 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:15:17.489 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:15:17.490 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:15:17.490 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:15:17.490 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:15:17.490 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:15:17.492 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:15:17.492 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:15:17.493 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:15:17.493 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:15:17.493 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:15:17.496 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:15:17.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:17.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:15:17.567 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:15:17.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:15:17.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:15:17.707 INFO:teuthology.orchestra.run.smithi163.stdout:client4930 2024-07-26T06:15:17.707 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:15:17.707 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:17.757 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:15:18.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 -- 172.21.15.163:0/1605307945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 msgr2=0x7f99dc0b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:18.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 --2- 172.21.15.163:0/1605307945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0b6bd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009830 tx=0x7f99d402a490 comp rx=0 tx=0).stop 2024-07-26T06:15:18.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 -- 172.21.15.163:0/1605307945 shutdown_connections 2024-07-26T06:15:18.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 --2- 172.21.15.163:0/1605307945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0b6bd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 -- 172.21.15.163:0/1605307945 >> 172.21.15.163:0/1605307945 conn(0x7f99dc0cfda0 msgr2=0x7f99dc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:18.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 -- 172.21.15.163:0/1605307945 shutdown_connections 2024-07-26T06:15:18.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.015+0000 7f99e3000640 1 -- 172.21.15.163:0/1605307945 wait complete. 2024-07-26T06:15:18.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.017+0000 7f99e3000640 1 Processor -- start 2024-07-26T06:15:18.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.017+0000 7f99e3000640 1 -- start start 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e3000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99dc0b1e80 0x7f99dc14a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e3000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e3000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99dc15d5a0 0x7f99dc1550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e3000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc0b5130 con 0x7f99dc0b23d0 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e3000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc0b4fb0 con 0x7f99dc0b1e80 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e3000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f99dc0b52b0 con 0x7f99dc15d5a0 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e1200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e1200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0c7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53452/0 (socket says 172.21.15.163:53452) 2024-07-26T06:15:18.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.018+0000 7f99e1200640 1 -- 172.21.15.163:0/3634214786 learned_addr learned my addr 172.21.15.163:0/3634214786 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:18.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1200640 1 -- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99dc15d5a0 msgr2=0x7f99dc1550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:18.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1200640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99dc15d5a0 0x7f99dc1550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1200640 1 -- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99dc0b1e80 msgr2=0x7f99dc14a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1c00640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99dc0b1e80 0x7f99dc14a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1200640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99dc0b1e80 0x7f99dc14a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1200640 1 -- 172.21.15.163:0/3634214786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99dc0c8700 con 0x7f99dc0b23d0 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1c00640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99dc0b1e80 0x7f99dc14a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e1200640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0c7f60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00c480 tx=0x7f99cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0052d0 con 0x7f99dc0b23d0 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99cc005470 con 0x7f99dc0b23d0 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc012870 con 0x7f99dc0b23d0 2024-07-26T06:15:18.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99dc0c89e0 con 0x7f99dc0b23d0 2024-07-26T06:15:18.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.019+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99dc0c8fa0 con 0x7f99dc0b23d0 2024-07-26T06:15:18.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.021+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99dc050910 con 0x7f99dc0b23d0 2024-07-26T06:15:18.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.022+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f99cc0035f0 con 0x7f99dc0b23d0 2024-07-26T06:15:18.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.023+0000 7f99d9600640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f99c0076480 0x7f99c0078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:18.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.023+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f99cc097780 con 0x7f99dc0b23d0 2024-07-26T06:15:18.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.024+0000 7f99e1c00640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f99c0076480 0x7f99c0078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:18.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.025+0000 7f99e1c00640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f99c0076480 0x7f99c0078940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f99d0004a30 tx=0x7f99d0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:18.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.027+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99dc050910 con 0x7f99dc0b23d0 2024-07-26T06:15:18.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.276+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f99dc0b57b0 con 0x7f99dc0b23d0 2024-07-26T06:15:18.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.276+0000 7f99d9600640 1 -- 172.21.15.163:0/3634214786 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f99cc060e10 con 0x7f99dc0b23d0 2024-07-26T06:15:18.278 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f99c0076480 msgr2=0x7f99c0078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f99c0076480 0x7f99c0078940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f99d0004a30 tx=0x7f99d0009040 comp rx=0 tx=0).stop 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 msgr2=0x7f99dc0c7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0c7f60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00c480 tx=0x7f99cc00c940 comp rx=0 tx=0).stop 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 shutdown_connections 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f99c0076480 0x7f99c0078940 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99dc15d5a0 0x7f99dc1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99dc0b23d0 0x7f99dc0c7f60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 --2- 172.21.15.163:0/3634214786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99dc0b1e80 0x7f99dc14a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:18.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.279+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 >> 172.21.15.163:0/3634214786 conn(0x7f99dc0cfda0 msgr2=0x7f99dc15aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:18.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.280+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 shutdown_connections 2024-07-26T06:15:18.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:18.280+0000 7f99e3000640 1 -- 172.21.15.163:0/3634214786 wait complete. 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:15:18.300 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:15:18.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:18.301 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:15:18.360 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:15:18.360 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:18.361 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:18.362 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42465sec preferred_lft 42465sec 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.363 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::58b1:99ff:fee0:4877/64 scope link 2024-07-26T06:15:18.364 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:15:18.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:18.366 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:15:18.448 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:15:18.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:18.452 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:15:18.505 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:15:18.506 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:15:18.506 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:15:18.506 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:15:18.506 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:15:18.506 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:15:18.506 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:15:18.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:15:18.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:15:18.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:15:18.507 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:15:18.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:15:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:15:18.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:15:18.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:18.713 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:15:18.759 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:15:18.759 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:15:18.760 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:15:18.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:18.760 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:15:18.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:15:18.841 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:15:18.892 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:15:18.892 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:15:18.892 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:15:18.941 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:15:18.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:18.941 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:15:19.057 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:15:19.057 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:15:19.057 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:15:19.078 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:15:19.079 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:15:19.079 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:15:19.079 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:15:19.079 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:15:19.107 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:15:19.109 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:15:19.109 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:15:19.110 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:15:19.110 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:15:19.110 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:15:19.113 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:15:19.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:19.113 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:15:19.180 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:15:19.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:15:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:15:19.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:15:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:15:19.296 INFO:teuthology.orchestra.run.smithi164.stdout:client5406 2024-07-26T06:15:19.297 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:15:19.297 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:19.349 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-07-26T06:15:19.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.564+0000 7f698b200640 1 -- 172.21.15.163:0/364472371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 msgr2=0x7f698c0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:19.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.564+0000 7f698b200640 1 --2- 172.21.15.163:0/364472371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0b64c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f697c009830 tx=0x7f697c02a3f0 comp rx=0 tx=0).stop 2024-07-26T06:15:19.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.565+0000 7f698b200640 1 -- 172.21.15.163:0/364472371 shutdown_connections 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.565+0000 7f698b200640 1 --2- 172.21.15.163:0/364472371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0b64c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.565+0000 7f698b200640 1 --2- 172.21.15.163:0/364472371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f698c15d5a0 0x7f698c0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.565+0000 7f698b200640 1 -- 172.21.15.163:0/364472371 >> 172.21.15.163:0/364472371 conn(0x7f698c0cfda0 msgr2=0x7f698c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.565+0000 7f698b200640 1 -- 172.21.15.163:0/364472371 shutdown_connections 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.565+0000 7f698b200640 1 -- 172.21.15.163:0/364472371 wait complete. 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.566+0000 7f698b200640 1 Processor -- start 2024-07-26T06:15:19.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.567+0000 7f698b200640 1 -- start start 2024-07-26T06:15:19.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.567+0000 7f698b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f698c0b1e80 0x7f698c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:19.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f698b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:19.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f698b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f698c15d5a0 0x7f698c0b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:19.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f698b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b5130 con 0x7f698c0ce830 2024-07-26T06:15:19.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f698b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b4fb0 con 0x7f698c0b1e80 2024-07-26T06:15:19.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f698b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f698c0b52b0 con 0x7f698c15d5a0 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53472/0 (socket says 172.21.15.163:53472) 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 -- 172.21.15.163:0/1593794018 learned_addr learned my addr 172.21.15.163:0/1593794018 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989e00640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f698c0b1e80 0x7f698c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 -- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f698c15d5a0 msgr2=0x7f698c0b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f698c15d5a0 0x7f698c0b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 -- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f698c0b1e80 msgr2=0x7f698c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f698c0b1e80 0x7f698c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989400640 1 -- 172.21.15.163:0/1593794018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c0c6eb0 con 0x7f698c0ce830 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.568+0000 7f6989e00640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f698c0b1e80 0x7f698c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.569+0000 7f6989400640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0cbb10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f697c009950 tx=0x7f697c032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:19.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.569+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c040070 con 0x7f698c0ce830 2024-07-26T06:15:19.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.569+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f697c032e70 con 0x7f698c0ce830 2024-07-26T06:15:19.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.569+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c003d90 con 0x7f698c0ce830 2024-07-26T06:15:19.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.569+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c0c7150 con 0x7f698c0ce830 2024-07-26T06:15:19.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.569+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c0c7710 con 0x7f698c0ce830 2024-07-26T06:15:19.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.570+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c0cf5a0 con 0x7f698c0ce830 2024-07-26T06:15:19.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.571+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f697c039040 con 0x7f698c0ce830 2024-07-26T06:15:19.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.573+0000 7f6982000640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6968076480 0x7f6968078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:19.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.573+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f697c0b91d0 con 0x7f698c0ce830 2024-07-26T06:15:19.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.573+0000 7f6989e00640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6968076480 0x7f6968078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.575+0000 7f6989e00640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6968076480 0x7f6968078940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f698c0d19b0 tx=0x7f6974005dd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.578+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f698c0cf5a0 con 0x7f698c0ce830 2024-07-26T06:15:19.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:19.833+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f698c0ce2f0 con 0x7f698c0ce830 2024-07-26T06:15:20.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.750+0000 7f6982000640 1 -- 172.21.15.163:0/1593794018 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v26) ==== 136+0+0 (secure 0 0 0) 0x7f697c082750 con 0x7f698c0ce830 2024-07-26T06:15:20.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6968076480 msgr2=0x7f6968078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:20.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6968076480 0x7f6968078940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f698c0d19b0 tx=0x7f6974005dd0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 msgr2=0x7f698c0cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0cbb10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f697c009950 tx=0x7f697c032c80 comp rx=0 tx=0).stop 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 shutdown_connections 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6968076480 0x7f6968078940 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f698c15d5a0 0x7f698c0b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f698c0ce830 0x7f698c0cbb10 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 --2- 172.21.15.163:0/1593794018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f698c0b1e80 0x7f698c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.754+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 >> 172.21.15.163:0/1593794018 conn(0x7f698c0cfda0 msgr2=0x7f698c159430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.755+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 shutdown_connections 2024-07-26T06:15:20.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.755+0000 7f698b200640 1 -- 172.21.15.163:0/1593794018 wait complete. 2024-07-26T06:15:20.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-07-26T06:15:20.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 -- 172.21.15.163:0/932677465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 msgr2=0x7fc71806c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:20.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 --2- 172.21.15.163:0/932677465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71806c3b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc718069760 tx=0x7fc71002a980 comp rx=0 tx=0).stop 2024-07-26T06:15:20.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 -- 172.21.15.163:0/932677465 shutdown_connections 2024-07-26T06:15:20.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 --2- 172.21.15.163:0/932677465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71806c3b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 --2- 172.21.15.163:0/932677465 >> v2:172.21.15.164:3300/0 conn(0x7fc71806f350 0x7fc7180b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 -- 172.21.15.163:0/932677465 >> 172.21.15.163:0/932677465 conn(0x7fc71806d420 msgr2=0x7fc71806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:20.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 -- 172.21.15.163:0/932677465 shutdown_connections 2024-07-26T06:15:20.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.984+0000 7fc71d800640 1 -- 172.21.15.163:0/932677465 wait complete. 2024-07-26T06:15:20.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.986+0000 7fc71d800640 1 Processor -- start 2024-07-26T06:15:20.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.986+0000 7fc71d800640 1 -- start start 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc71d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc71d800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc71806f350 0x7fc7180ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc71d800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc718159380 0x7fc7180ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc71d800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc718164280 con 0x7fc71806bf40 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc71d800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7180cdc10 con 0x7fc71806f350 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc71d800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc718164400 con 0x7fc718159380 2024-07-26T06:15:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71815c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53498/0 (socket says 172.21.15.163:53498) 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 -- 172.21.15.163:0/238997736 learned_addr learned my addr 172.21.15.163:0/238997736 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc717400640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc718159380 0x7fc7180ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716000640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc71806f350 0x7fc7180ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 -- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc718159380 msgr2=0x7fc7180ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc718159380 0x7fc7180ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 -- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc71806f350 msgr2=0x7fc7180ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc71806f350 0x7fc7180ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.987+0000 7fc716a00640 1 -- 172.21.15.163:0/238997736 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71815c860 con 0x7fc71806bf40 2024-07-26T06:15:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.988+0000 7fc716a00640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71815c320 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc70800d770 tx=0x7fc70800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:20.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.988+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708017070 con 0x7fc71806bf40 2024-07-26T06:15:20.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.988+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc708009d20 con 0x7fc71806bf40 2024-07-26T06:15:20.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.988+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc718072340 con 0x7fc71806bf40 2024-07-26T06:15:20.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.988+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708005280 con 0x7fc71806bf40 2024-07-26T06:15:20.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.988+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc718072900 con 0x7fc71806bf40 2024-07-26T06:15:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.989+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71806c3b0 con 0x7fc71806bf40 2024-07-26T06:15:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.990+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fc708005420 con 0x7fc71806bf40 2024-07-26T06:15:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.991+0000 7fc70ea00640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6f8076480 0x7fc6f8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.991+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fc708097d20 con 0x7fc71806bf40 2024-07-26T06:15:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.992+0000 7fc716000640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6f8076480 0x7fc6f8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.993+0000 7fc716000640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6f8076480 0x7fc6f8078940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc704006fd0 tx=0x7fc704008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:20.996+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc7080612a0 con 0x7fc71806bf40 2024-07-26T06:15:21.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.222+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fc718156ef0 con 0x7fc6f8076480 2024-07-26T06:15:21.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.227+0000 7fc70ea00640 1 -- 172.21.15.163:0/238997736 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fc718156ef0 con 0x7fc6f8076480 2024-07-26T06:15:21.227 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:15:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:15:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephfs" 2024-07-26T06:15:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:15:21.228 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:15:21.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6f8076480 msgr2=0x7fc6f8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6f8076480 0x7fc6f8078940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc704006fd0 tx=0x7fc704008040 comp rx=0 tx=0).stop 2024-07-26T06:15:21.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 msgr2=0x7fc71815c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71815c320 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc70800d770 tx=0x7fc70800dc30 comp rx=0 tx=0).stop 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 shutdown_connections 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc6f8076480 0x7fc6f8078940 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc718159380 0x7fc7180ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc71806f350 0x7fc7180ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 --2- 172.21.15.163:0/238997736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc71806bf40 0x7fc71815c320 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.229+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 >> 172.21.15.163:0/238997736 conn(0x7fc71806d420 msgr2=0x7fc718159c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.230+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 shutdown_connections 2024-07-26T06:15:21.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.230+0000 7fc71d800640 1 -- 172.21.15.163:0/238997736 wait complete. 2024-07-26T06:15:21.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:15:21.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.439+0000 7fbbfea00640 1 -- 172.21.15.163:0/2517218357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b1e80 msgr2=0x7fbbf80b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.439+0000 7fbbfea00640 1 --2- 172.21.15.163:0/2517218357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b1e80 0x7fbbf80b64c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbbf806f160 tx=0x7fbbe802a8f0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.439+0000 7fbbfea00640 1 -- 172.21.15.163:0/2517218357 shutdown_connections 2024-07-26T06:15:21.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.439+0000 7fbbfea00640 1 --2- 172.21.15.163:0/2517218357 >> v2:172.21.15.164:3300/0 conn(0x7fbbf80b23d0 0x7fbbf80b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.440+0000 7fbbfea00640 1 --2- 172.21.15.163:0/2517218357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b1e80 0x7fbbf80b64c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.440+0000 7fbbfea00640 1 -- 172.21.15.163:0/2517218357 >> 172.21.15.163:0/2517218357 conn(0x7fbbf80cfda0 msgr2=0x7fbbf80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:21.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.440+0000 7fbbfea00640 1 -- 172.21.15.163:0/2517218357 shutdown_connections 2024-07-26T06:15:21.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.440+0000 7fbbfea00640 1 -- 172.21.15.163:0/2517218357 wait complete. 2024-07-26T06:15:21.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.441+0000 7fbbfea00640 1 Processor -- start 2024-07-26T06:15:21.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.441+0000 7fbbfea00640 1 -- start start 2024-07-26T06:15:21.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.442+0000 7fbbfea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbf80b1e80 0x7fbbf81507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.442+0000 7fbbfea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 0x7fbbf81f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.442+0000 7fbbfea00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbf815d5a0 0x7fbbf81f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.442+0000 7fbbfea00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf80b4f20 con 0x7fbbf80b23d0 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.442+0000 7fbbfea00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf80b4da0 con 0x7fbbf80b1e80 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.442+0000 7fbbfea00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf80b50a0 con 0x7fbbf815d5a0 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 0x7fbbf81f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 0x7fbbf81f4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53510/0 (socket says 172.21.15.163:53510) 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 -- 172.21.15.163:0/3057561083 learned_addr learned my addr 172.21.15.163:0/3057561083 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 -- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbf815d5a0 msgr2=0x7fbbf81f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbf815d5a0 0x7fbbf81f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 -- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbf80b1e80 msgr2=0x7fbbf81507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbf80b1e80 0x7fbbf81507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 -- 172.21.15.163:0/3057561083 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf81f5760 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.443+0000 7fbbfcc00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 0x7fbbf81f4b30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8033040 tx=0x7fbbe8003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.444+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8004240 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.444+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbe802d070 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.444+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8004680 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.444+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf8150e40 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.444+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf8151400 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.445+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf815b960 con 0x7fbbf80b23d0 2024-07-26T06:15:21.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.446+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fbbe8031480 con 0x7fbbf80b23d0 2024-07-26T06:15:21.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.447+0000 7fbbf5600640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbbdc076480 0x7fbbdc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:21.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.447+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fbbe80b9660 con 0x7fbbf80b23d0 2024-07-26T06:15:21.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.448+0000 7fbbfd600640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbbdc076480 0x7fbbdc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:21.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.449+0000 7fbbfd600640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbbdc076480 0x7fbbdc078940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009650 tx=0x7fbbf0006ba0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:21.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.451+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbbf80b6f40 con 0x7fbbf80b23d0 2024-07-26T06:15:21.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.653+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fbbf80b8f00 con 0x7fbbf80b23d0 2024-07-26T06:15:21.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.654+0000 7fbbf5600640 1 -- 172.21.15.163:0/3057561083 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7fbbe8004820 con 0x7fbbf80b23d0 2024-07-26T06:15:21.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbbdc076480 msgr2=0x7fbbdc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbbdc076480 0x7fbbdc078940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009650 tx=0x7fbbf0006ba0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 msgr2=0x7fbbf81f4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:21.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 0x7fbbf81f4b30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8033040 tx=0x7fbbe8003cd0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 shutdown_connections 2024-07-26T06:15:21.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbbdc076480 0x7fbbdc078940 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbf815d5a0 0x7fbbf81f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbf80b23d0 0x7fbbf81f4b30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 --2- 172.21.15.163:0/3057561083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbf80b1e80 0x7fbbf81507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:21.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 >> 172.21.15.163:0/3057561083 conn(0x7fbbf80cfda0 msgr2=0x7fbbf815bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:21.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 shutdown_connections 2024-07-26T06:15:21.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:21.657+0000 7fbbfea00640 1 -- 172.21.15.163:0/3057561083 wait complete. 2024-07-26T06:15:21.676 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:15:21.676 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:21.723 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:15:21.723 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:21.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:15:22.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 -- 172.21.15.163:0/2018803335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 msgr2=0x7f071c0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 --2- 172.21.15.163:0/2018803335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c0b64c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0704005fd0 tx=0x7f070402a8e0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 -- 172.21.15.163:0/2018803335 shutdown_connections 2024-07-26T06:15:22.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 --2- 172.21.15.163:0/2018803335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c0b64c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 --2- 172.21.15.163:0/2018803335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f071c15d5a0 0x7f071c0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 -- 172.21.15.163:0/2018803335 >> 172.21.15.163:0/2018803335 conn(0x7f071c0cfda0 msgr2=0x7f071c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:22.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 -- 172.21.15.163:0/2018803335 shutdown_connections 2024-07-26T06:15:22.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.016+0000 7f071b200640 1 -- 172.21.15.163:0/2018803335 wait complete. 2024-07-26T06:15:22.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.018+0000 7f071b200640 1 Processor -- start 2024-07-26T06:15:22.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.018+0000 7f071b200640 1 -- start start 2024-07-26T06:15:22.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f071c0b1e80 0x7f071c1550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c14d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f071c15d5a0 0x7f071c1478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f071c0b5130 con 0x7f071c0ce830 2024-07-26T06:15:22.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f071c0b4fb0 con 0x7f071c0b1e80 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f071c0b52b0 con 0x7f071c15d5a0 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f0719400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c14d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f0719400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c14d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53520/0 (socket says 172.21.15.163:53520) 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f0719400640 1 -- 172.21.15.163:0/3988844946 learned_addr learned my addr 172.21.15.163:0/3988844946 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f071a800640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f071c15d5a0 0x7f071c1478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.019+0000 7f0719e00640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f071c0b1e80 0x7f071c1550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719400640 1 -- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f071c15d5a0 msgr2=0x7f071c1478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719400640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f071c15d5a0 0x7f071c1478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719400640 1 -- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f071c0b1e80 msgr2=0x7f071c1550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719400640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f071c0b1e80 0x7f071c1550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719400640 1 -- 172.21.15.163:0/3988844946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c0c7630 con 0x7f071c0ce830 2024-07-26T06:15:22.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719e00640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f071c0b1e80 0x7f071c1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:22.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0719400640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c14d430 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0704009bf0 tx=0x7f0704003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:22.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704004240 con 0x7f071c0ce830 2024-07-26T06:15:22.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0704004400 con 0x7f071c0ce830 2024-07-26T06:15:22.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704004850 con 0x7f071c0ce830 2024-07-26T06:15:22.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.020+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c0c7910 con 0x7f071c0ce830 2024-07-26T06:15:22.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.021+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c0c7ed0 con 0x7f071c0ce830 2024-07-26T06:15:22.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.022+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071c0cf5a0 con 0x7f071c0ce830 2024-07-26T06:15:22.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.023+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f0704031470 con 0x7f071c0ce830 2024-07-26T06:15:22.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.024+0000 7f0712000640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f06f8076620 0x7f06f8078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.024+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f07040b9550 con 0x7f071c0ce830 2024-07-26T06:15:22.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.025+0000 7f0719e00640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f06f8076620 0x7f06f8078ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:22.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.026+0000 7f0719e00640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f06f8076620 0x7f06f8078ae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f070c0058c0 tx=0x7f070c005cb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:22.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.029+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f071c0cf5a0 con 0x7f071c0ce830 2024-07-26T06:15:22.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.249+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f071c0ce2f0 con 0x7f071c0ce830 2024-07-26T06:15:22.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.250+0000 7f0712000640 1 -- 172.21.15.163:0/3988844946 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v47) ==== 80+0+0 (secure 0 0 0) 0x7f0704082b50 con 0x7f071c0ce830 2024-07-26T06:15:22.251 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:15:22.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f06f8076620 msgr2=0x7f06f8078ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f06f8076620 0x7f06f8078ae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f070c0058c0 tx=0x7f070c005cb0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 msgr2=0x7f071c14d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c14d430 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0704009bf0 tx=0x7f0704003f90 comp rx=0 tx=0).stop 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 shutdown_connections 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f06f8076620 0x7f06f8078ae0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f071c15d5a0 0x7f071c1478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f071c0ce830 0x7f071c14d430 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 --2- 172.21.15.163:0/3988844946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f071c0b1e80 0x7f071c1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.253+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 >> 172.21.15.163:0/3988844946 conn(0x7f071c0cfda0 msgr2=0x7f071c15b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.254+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 shutdown_connections 2024-07-26T06:15:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.254+0000 7f071b200640 1 -- 172.21.15.163:0/3988844946 wait complete. 2024-07-26T06:15:22.274 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:15:22.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:22.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-07-26T06:15:22.366 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:15:22.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:22.367 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:15:22.418 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:15:22.419 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:22.475 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:15:22.475 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:15:22.519 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:15:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.734+0000 7f7954a00640 1 -- 172.21.15.163:0/101635021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 msgr2=0x7f79500b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.735+0000 7f7954a00640 1 --2- 172.21.15.163:0/101635021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500b4f70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7938009830 tx=0x7f793802a480 comp rx=0 tx=0).stop 2024-07-26T06:15:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.735+0000 7f7954a00640 1 -- 172.21.15.163:0/101635021 shutdown_connections 2024-07-26T06:15:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.735+0000 7f7954a00640 1 --2- 172.21.15.163:0/101635021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500b4f70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.735+0000 7f7954a00640 1 -- 172.21.15.163:0/101635021 >> 172.21.15.163:0/101635021 conn(0x7f79500b3060 msgr2=0x7f79500b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.735+0000 7f7954a00640 1 -- 172.21.15.163:0/101635021 shutdown_connections 2024-07-26T06:15:22.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.735+0000 7f7954a00640 1 -- 172.21.15.163:0/101635021 wait complete. 2024-07-26T06:15:22.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.737+0000 7f7954a00640 1 Processor -- start 2024-07-26T06:15:22.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.737+0000 7f7954a00640 1 -- start start 2024-07-26T06:15:22.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f7954a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f7954a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79500cff20 0x7f795006d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f7954a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f795015b0c0 0x7f795006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f7954a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79500b5e20 con 0x7f7950071130 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f7954a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f79500b5ca0 con 0x7f795015b0c0 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f7954a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f79500b5fa0 con 0x7f79500cff20 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53526/0 (socket says 172.21.15.163:53526) 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 -- 172.21.15.163:0/596556625 learned_addr learned my addr 172.21.15.163:0/596556625 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 -- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79500cff20 msgr2=0x7f795006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79500cff20 0x7f795006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 -- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f795015b0c0 msgr2=0x7f795006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f795015b0c0 0x7f795006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.738+0000 7f794e800640 1 -- 172.21.15.163:0/596556625 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795006e9a0 con 0x7f7950071130 2024-07-26T06:15:22.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.739+0000 7f794e800640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500ca6e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f794000c460 tx=0x7f794000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.739+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940016020 con 0x7f7950071130 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.739+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7940012940 con 0x7f7950071130 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.739+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940012d90 con 0x7f7950071130 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.739+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7950159060 con 0x7f7950071130 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.739+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f795006e3e0 con 0x7f7950071130 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.740+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7950159230 con 0x7f7950071130 2024-07-26T06:15:22.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.741+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f7940019020 con 0x7f7950071130 2024-07-26T06:15:22.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.742+0000 7f7947400640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f792c076480 0x7f792c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:22.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.744+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f79400977b0 con 0x7f7950071130 2024-07-26T06:15:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.744+0000 7f7945e00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f792c076480 0x7f792c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.744+0000 7f7945e00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f792c076480 0x7f792c078940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7938034000 tx=0x7f79380046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.747+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7940020030 con 0x7f7950071130 2024-07-26T06:15:22.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.968+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7950070ca0 con 0x7f7950071130 2024-07-26T06:15:22.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.969+0000 7f7947400640 1 -- 172.21.15.163:0/596556625 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v47) ==== 80+0+0 (secure 0 0 0) 0x7f7940060e40 con 0x7f7950071130 2024-07-26T06:15:22.969 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:15:22.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f792c076480 msgr2=0x7f792c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f792c076480 0x7f792c078940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7938034000 tx=0x7f79380046c0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 msgr2=0x7f79500ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500ca6e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f794000c460 tx=0x7f794000c920 comp rx=0 tx=0).stop 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 shutdown_connections 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f792c076480 0x7f792c078940 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f795015b0c0 0x7f795006d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79500cff20 0x7f795006d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 --2- 172.21.15.163:0/596556625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7950071130 0x7f79500ca6e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:22.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 >> 172.21.15.163:0/596556625 conn(0x7f79500b3060 msgr2=0x7f795015e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:22.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 shutdown_connections 2024-07-26T06:15:22.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:22.972+0000 7f7954a00640 1 -- 172.21.15.163:0/596556625 wait complete. 2024-07-26T06:15:22.992 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:15:22.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:22.992 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-07-26T06:15:23.093 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:15:23.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:15:23.093 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:15:23.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:23.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 -- 172.21.15.163:0/4167860914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c071c40 msgr2=0x7efd0c0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:23.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 --2- 172.21.15.163:0/4167860914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c071c40 0x7efd0c0b3710 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efd0c06ce80 tx=0x7efd0402aac0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 -- 172.21.15.163:0/4167860914 shutdown_connections 2024-07-26T06:15:23.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 --2- 172.21.15.163:0/4167860914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c071c40 0x7efd0c0b3710 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 --2- 172.21.15.163:0/4167860914 >> v2:172.21.15.164:3300/0 conn(0x7efd0c1556f0 0x7efd0c0cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 -- 172.21.15.163:0/4167860914 >> 172.21.15.163:0/4167860914 conn(0x7efd0c0b0e30 msgr2=0x7efd0c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:23.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 -- 172.21.15.163:0/4167860914 shutdown_connections 2024-07-26T06:15:23.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.352+0000 7efd0b200640 1 -- 172.21.15.163:0/4167860914 wait complete. 2024-07-26T06:15:23.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.354+0000 7efd0b200640 1 Processor -- start 2024-07-26T06:15:23.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.354+0000 7efd0b200640 1 -- start start 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0c071c40 0x7efd0c1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 0x7efd0c14e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd0c1556f0 0x7efd0c153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0b3fe0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0b3e60 con 0x7efd0c1556f0 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd0c0b4160 con 0x7efd0c071c40 2024-07-26T06:15:23.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd03e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 0x7efd0c14e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd03e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 0x7efd0c14e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53544/0 (socket says 172.21.15.163:53544) 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd03e00640 1 -- 172.21.15.163:0/3789523488 learned_addr learned my addr 172.21.15.163:0/3789523488 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd09e00640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0c071c40 0x7efd0c1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd0a800640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd0c1556f0 0x7efd0c153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd03e00640 1 -- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0c071c40 msgr2=0x7efd0c1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.356+0000 7efd03e00640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0c071c40 0x7efd0c1525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd03e00640 1 -- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd0c1556f0 msgr2=0x7efd0c153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd03e00640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd0c1556f0 0x7efd0c153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd03e00640 1 -- 172.21.15.163:0/3789523488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c14e830 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd03e00640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 0x7efd0c14e2f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c480 tx=0x7efcf400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:23.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf40052d0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcf4005470 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4012870 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0c1541f0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0c152ef0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.358+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0c0b46f0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.360+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7efcf40035f0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.361+0000 7efd02000640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce8076480 0x7efce8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:23.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.361+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7efcf40977c0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.361+0000 7efd09e00640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce8076480 0x7efce8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:23.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.362+0000 7efd09e00640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce8076480 0x7efce8078940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0058c0 tx=0x7efcfc00c440 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:23.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.365+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd0c1541f0 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.616+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd0c0ced60 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.617+0000 7efd02000640 1 -- 172.21.15.163:0/3789523488 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7efd0c14e830 con 0x7efd0c0ce2d0 2024-07-26T06:15:23.618 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:23.618 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:23.618 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.620+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce8076480 msgr2=0x7efce8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.620+0000 7efd0b200640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce8076480 0x7efce8078940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0058c0 tx=0x7efcfc00c440 comp rx=0 tx=0).stop 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.620+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 msgr2=0x7efd0c14e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.620+0000 7efd0b200640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 0x7efd0c14e2f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c480 tx=0x7efcf400c940 comp rx=0 tx=0).stop 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 shutdown_connections 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7efce8076480 0x7efce8078940 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd0c1556f0 0x7efd0c153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd0c0ce2d0 0x7efd0c14e2f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 --2- 172.21.15.163:0/3789523488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd0c071c40 0x7efd0c1525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 >> 172.21.15.163:0/3789523488 conn(0x7efd0c0b0e30 msgr2=0x7efd0c15df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:23.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 shutdown_connections 2024-07-26T06:15:23.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:23.621+0000 7efd0b200640 1 -- 172.21.15.163:0/3789523488 wait complete. 2024-07-26T06:15:23.641 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:23.875 INFO:teuthology.orchestra.run.smithi164.stdout:[] 2024-07-26T06:15:23.875 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-07-26T06:15:23.876 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol2390 2024-07-26T06:15:24.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff7200640 1 -- 172.21.15.163:0/1648761275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfdc004800 msgr2=0x7fbfdc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff7200640 1 --2- 172.21.15.163:0/1648761275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfdc004800 0x7fbfdc004c80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbff808c7d0 tx=0x7fbfdc005830 comp rx=0 tx=0).stop 2024-07-26T06:15:24.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff6800640 1 -- 172.21.15.163:0/1648761275 reap_dead start 2024-07-26T06:15:24.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff7200640 1 -- 172.21.15.163:0/1648761275 shutdown_connections 2024-07-26T06:15:24.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff7200640 1 -- 172.21.15.163:0/1648761275 >> 172.21.15.163:0/1648761275 conn(0x7fbff80cfd30 msgr2=0x7fbff80d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:24.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff7200640 1 -- 172.21.15.163:0/1648761275 shutdown_connections 2024-07-26T06:15:24.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.093+0000 7fbff7200640 1 -- 172.21.15.163:0/1648761275 wait complete. 2024-07-26T06:15:24.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.095+0000 7fbff7200640 1 Processor -- start 2024-07-26T06:15:24.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.095+0000 7fbff7200640 1 -- start start 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff7200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc004800 0x7fbff8156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff7200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 0x7fbff80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff7200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbff814f250 0x7fbff814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff7200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbff80b4d30 con 0x7fbff81533b0 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff7200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbff815d900 con 0x7fbff814f250 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff7200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbff80b4eb0 con 0x7fbfdc004800 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.096+0000 7fbff5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 0x7fbff80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 0x7fbff80b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53572/0 (socket says 172.21.15.163:53572) 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 -- 172.21.15.163:0/3524111241 learned_addr learned my addr 172.21.15.163:0/3524111241 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5e00640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc004800 0x7fbff8156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff6800640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbff814f250 0x7fbff814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 -- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc004800 msgr2=0x7fbff8156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc004800 0x7fbff8156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 -- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbff814f250 msgr2=0x7fbff814f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbff814f250 0x7fbff814f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 -- 172.21.15.163:0/3524111241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff814fc20 con 0x7fbff81533b0 2024-07-26T06:15:24.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.097+0000 7fbff5400640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 0x7fbff80b3640 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbff8050dc0 tx=0x7fbfe002aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:24.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.098+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0038070 con 0x7fbff81533b0 2024-07-26T06:15:24.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.098+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfe00035f0 con 0x7fbff81533b0 2024-07-26T06:15:24.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.098+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81538b0 con 0x7fbff81533b0 2024-07-26T06:15:24.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.098+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0003a00 con 0x7fbff81533b0 2024-07-26T06:15:24.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.098+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff8153e70 con 0x7fbff81533b0 2024-07-26T06:15:24.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.099+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff80b90c0 con 0x7fbff81533b0 2024-07-26T06:15:24.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.100+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7fbfe0030070 con 0x7fbff81533b0 2024-07-26T06:15:24.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.102+0000 7fbfee000640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbfd4076480 0x7fbfd4078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.102+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fbfe00b7300 con 0x7fbff81533b0 2024-07-26T06:15:24.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.102+0000 7fbff5e00640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbfd4076480 0x7fbfd4078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.103+0000 7fbff5e00640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbfd4076480 0x7fbfd4078940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8005fd0 tx=0x7fbfe8009500 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:24.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.106+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbff80b90c0 con 0x7fbff81533b0 2024-07-26T06:15:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.342+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol2390", "target": ["mon-mgr", ""]}) -- 0x7fbff80cf900 con 0x7fbfd4076480 2024-07-26T06:15:24.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:15:24.354+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:15:24.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:15:24.354+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:15:24.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:15:24.354+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:15:24.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:15:24.354+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:15:24.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:15:24.354+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:15:24.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.427+0000 7fbfee000640 1 -- 172.21.15.163:0/3524111241 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbff80cf900 con 0x7fbfd4076480 2024-07-26T06:15:24.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.429+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbfd4076480 msgr2=0x7fbfd4078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.429+0000 7fbff7200640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbfd4076480 0x7fbfd4078940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8005fd0 tx=0x7fbfe8009500 comp rx=0 tx=0).stop 2024-07-26T06:15:24.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.429+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 msgr2=0x7fbff80b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.429+0000 7fbff7200640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 0x7fbff80b3640 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbff8050dc0 tx=0x7fbfe002aec0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 shutdown_connections 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fbfd4076480 0x7fbfd4078940 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbff814f250 0x7fbff814f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbff81533b0 0x7fbff80b3640 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 --2- 172.21.15.163:0/3524111241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc004800 0x7fbff8156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 >> 172.21.15.163:0/3524111241 conn(0x7fbff80cfd30 msgr2=0x7fbff80cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 shutdown_connections 2024-07-26T06:15:24.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.430+0000 7fbff7200640 1 -- 172.21.15.163:0/3524111241 wait complete. 2024-07-26T06:15:24.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:24.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 -- 172.21.15.163:0/2918826411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42841556f0 msgr2=0x7f42840cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 --2- 172.21.15.163:0/2918826411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42841556f0 0x7f42840cdd90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4274009830 tx=0x7f427402a510 comp rx=0 tx=0).stop 2024-07-26T06:15:24.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 -- 172.21.15.163:0/2918826411 shutdown_connections 2024-07-26T06:15:24.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 --2- 172.21.15.163:0/2918826411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42841556f0 0x7f42840cdd90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 -- 172.21.15.163:0/2918826411 >> 172.21.15.163:0/2918826411 conn(0x7f42840b0e30 msgr2=0x7f42840b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:24.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 -- 172.21.15.163:0/2918826411 shutdown_connections 2024-07-26T06:15:24.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.640+0000 7f4289600640 1 -- 172.21.15.163:0/2918826411 wait complete. 2024-07-26T06:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.642+0000 7f4289600640 1 Processor -- start 2024-07-26T06:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.642+0000 7f4289600640 1 -- start start 2024-07-26T06:15:24.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f4289600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284071c40 0x7f42841458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f4289600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 0x7f42840c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f4289600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42841556f0 0x7f42840cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f4289600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42840b41f0 con 0x7f42840ce2d0 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f4289600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42840b4070 con 0x7f42841556f0 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f4289600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42840b4370 con 0x7f4284071c40 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f427be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 0x7f42840c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f427be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 0x7f42840c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:53580/0 (socket says 172.21.15.163:53580) 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.643+0000 7f427be00640 1 -- 172.21.15.163:0/3110010537 learned_addr learned my addr 172.21.15.163:0/3110010537 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f4283e00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284071c40 0x7f42841458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f427be00640 1 -- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284071c40 msgr2=0x7f42841458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f427be00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284071c40 0x7f42841458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f427be00640 1 -- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42841556f0 msgr2=0x7f42840cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f4288c00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42841556f0 0x7f42840cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f427be00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42841556f0 0x7f42840cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f427be00640 1 -- 172.21.15.163:0/3110010537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42840c8700 con 0x7f42840ce2d0 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f427be00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 0x7f42840c97a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f426c00c500 tx=0x7f426c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.644+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c017070 con 0x7f42840ce2d0 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.645+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f426c0035f0 con 0x7f42840ce2d0 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.645+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42840c89e0 con 0x7f42840ce2d0 2024-07-26T06:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.645+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c003a40 con 0x7f42840ce2d0 2024-07-26T06:15:24.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.645+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42840c8fa0 con 0x7f42840ce2d0 2024-07-26T06:15:24.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.646+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42840b3270 con 0x7f42840ce2d0 2024-07-26T06:15:24.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.647+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f426c01a070 con 0x7f42840ce2d0 2024-07-26T06:15:24.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.648+0000 7f4280c00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4260076480 0x7f4260078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:24.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.648+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f426c096ff0 con 0x7f42840ce2d0 2024-07-26T06:15:24.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.649+0000 7f4283e00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4260076480 0x7f4260078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:24.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.650+0000 7f4283e00640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4260076480 0x7f4260078940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4274034000 tx=0x7f4274004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:24.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.652+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f42840b3270 con 0x7f42840ce2d0 2024-07-26T06:15:24.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.892+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42840b34f0 con 0x7f42840ce2d0 2024-07-26T06:15:24.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.893+0000 7f4280c00640 1 -- 172.21.15.163:0/3110010537 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f426c0606c0 con 0x7f42840ce2d0 2024-07-26T06:15:24.894 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:24.894 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:24.895 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:24.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4260076480 msgr2=0x7f4260078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4260076480 0x7f4260078940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4274034000 tx=0x7f4274004730 comp rx=0 tx=0).stop 2024-07-26T06:15:24.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 msgr2=0x7f42840c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 0x7f42840c97a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f426c00c500 tx=0x7f426c00c9c0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 shutdown_connections 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4260076480 0x7f4260078940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42841556f0 0x7f42840cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42840ce2d0 0x7f42840c97a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 --2- 172.21.15.163:0/3110010537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284071c40 0x7f42841458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 >> 172.21.15.163:0/3110010537 conn(0x7f42840b0e30 msgr2=0x7f4284150bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 shutdown_connections 2024-07-26T06:15:24.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:24.896+0000 7f4289600640 1 -- 172.21.15.163:0/3110010537 wait complete. 2024-07-26T06:15:24.915 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:25.147 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.77098710599999998,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.961977459796195,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:25.147 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.961977459796195 }, "state": "open", "uptime": 0.770987106 } ] 2024-07-26T06:15:25.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:25.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 -- 172.21.15.163:0/2379451945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 msgr2=0x7f969006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:25.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 --2- 172.21.15.163:0/2379451945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f969006bf40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9690069760 tx=0x7f967802a560 comp rx=0 tx=0).stop 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 -- 172.21.15.163:0/2379451945 shutdown_connections 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 --2- 172.21.15.163:0/2379451945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f969006c480 0x7f969015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 --2- 172.21.15.163:0/2379451945 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f969006bf40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 --2- 172.21.15.163:0/2379451945 >> v2:172.21.15.164:3300/0 conn(0x7f9690159380 0x7f96900b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 -- 172.21.15.163:0/2379451945 >> 172.21.15.163:0/2379451945 conn(0x7f969006d420 msgr2=0x7f969006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 -- 172.21.15.163:0/2379451945 shutdown_connections 2024-07-26T06:15:25.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.358+0000 7f9695600640 1 -- 172.21.15.163:0/2379451945 wait complete. 2024-07-26T06:15:25.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.360+0000 7f9695600640 1 Processor -- start 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.360+0000 7f9695600640 1 -- start start 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9695600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969006c480 0x7f96901427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9695600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f9690142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9695600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9690159380 0x7f9690150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9695600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f969015d430 con 0x7f96900b40a0 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9695600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f969006f580 con 0x7f969006c480 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9695600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f969015d5b0 con 0x7f9690159380 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9687e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f9690142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9687e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f9690142ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33872/0 (socket says 172.21.15.163:33872) 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.361+0000 7f9687e00640 1 -- 172.21.15.163:0/2133031650 learned_addr learned my addr 172.21.15.163:0/2133031650 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:25.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9687e00640 1 -- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9690159380 msgr2=0x7f9690150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f968f400640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9690159380 0x7f9690150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f968ea00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969006c480 0x7f96901427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9687e00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9690159380 0x7f9690150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9687e00640 1 -- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969006c480 msgr2=0x7f96901427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9687e00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969006c480 0x7f96901427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9687e00640 1 -- 172.21.15.163:0/2133031650 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9690164b20 con 0x7f96900b40a0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f968f400640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9690159380 0x7f9690150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9687e00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f9690142ce0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9678009850 tx=0x7f9678003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967803f070 con 0x7f96900b40a0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96780047a0 con 0x7f96900b40a0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9690164e00 con 0x7f96900b40a0 2024-07-26T06:15:25.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9678035030 con 0x7f96900b40a0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.362+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96901653c0 con 0x7f96900b40a0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.364+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f969015da60 con 0x7f96900b40a0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.365+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 7) ==== 98354+0+0 (secure 0 0 0) 0x7f9678042070 con 0x7f96900b40a0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.365+0000 7f9686a00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f966c076480 0x7f966c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.366+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f96780b8b50 con 0x7f96900b40a0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.366+0000 7f968ea00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f966c076480 0x7f966c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.367+0000 7f968ea00640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f966c076480 0x7f966c078940 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9680005fd0 tx=0x7f9680005bc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:25.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.370+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9678081fb0 con 0x7f96900b40a0 2024-07-26T06:15:25.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.623+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96900b2520 con 0x7f96900b40a0 2024-07-26T06:15:25.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.624+0000 7f9686a00640 1 -- 172.21.15.163:0/2133031650 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f96780818b0 con 0x7f96900b40a0 2024-07-26T06:15:25.624 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:25.624 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:25.625 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:25.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f966c076480 msgr2=0x7f966c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:25.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f966c076480 0x7f966c078940 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9680005fd0 tx=0x7f9680005bc0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 msgr2=0x7f9690142ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f9690142ce0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9678009850 tx=0x7f9678003910 comp rx=0 tx=0).stop 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 shutdown_connections 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f966c076480 0x7f966c078940 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9690159380 0x7f9690150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f96900b40a0 0x7f9690142ce0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 --2- 172.21.15.163:0/2133031650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969006c480 0x7f96901427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.627+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 >> 172.21.15.163:0/2133031650 conn(0x7f969006d420 msgr2=0x7f96900ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.628+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 shutdown_connections 2024-07-26T06:15:25.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:25.628+0000 7f9695600640 1 -- 172.21.15.163:0/2133031650 wait complete. 2024-07-26T06:15:25.648 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:25.887 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":1.509974747,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.924511047034791,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:25.888 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.92451104703479 }, "state": "open", "uptime": 1.509974747 } ] 2024-07-26T06:15:25.888 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:15:26.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:15:26.157 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:15:30.889 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.105+0000 7f70cb200640 1 -- 172.21.15.163:0/974778327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b0003a80 msgr2=0x7f70b0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.105+0000 7f70cb200640 1 --2- 172.21.15.163:0/974778327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b0003a80 0x7f70b0003ea0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f70bc001dc0 tx=0x7f70b0005830 comp rx=0 tx=0).stop 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.105+0000 7f70ca800640 1 -- 172.21.15.163:0/974778327 reap_dead start 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.105+0000 7f70cb200640 1 -- 172.21.15.163:0/974778327 shutdown_connections 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.105+0000 7f70cb200640 1 -- 172.21.15.163:0/974778327 >> 172.21.15.163:0/974778327 conn(0x7f70cc0b0e30 msgr2=0x7f70cc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.105+0000 7f70cb200640 1 -- 172.21.15.163:0/974778327 shutdown_connections 2024-07-26T06:15:31.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.106+0000 7f70cb200640 1 -- 172.21.15.163:0/974778327 wait complete. 2024-07-26T06:15:31.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.107+0000 7f70cb200640 1 Processor -- start 2024-07-26T06:15:31.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.107+0000 7f70cb200640 1 -- start start 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.108+0000 7f70cb200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b0003a80 0x7f70cc14fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.108+0000 7f70cb200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 0x7f70cc15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.108+0000 7f70cb200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70cc15d6a0 0x7f70cc1e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.108+0000 7f70cb200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc0718c0 con 0x7f70cc1ea270 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.108+0000 7f70cb200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc071740 con 0x7f70b0003a80 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.108+0000 7f70cb200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f70cc0b3e20 con 0x7f70cc15d6a0 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 0x7f70cc15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:31.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 0x7f70cc15d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33884/0 (socket says 172.21.15.163:33884) 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 -- 172.21.15.163:0/2760627251 learned_addr learned my addr 172.21.15.163:0/2760627251 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70ca800640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70cc15d6a0 0x7f70cc1e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9e00640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b0003a80 0x7f70cc14fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 -- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70cc15d6a0 msgr2=0x7f70cc1e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70cc15d6a0 0x7f70cc1e6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 -- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b0003a80 msgr2=0x7f70cc14fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b0003a80 0x7f70cc14fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:31.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 -- 172.21.15.163:0/2760627251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70cc1e46f0 con 0x7f70cc1ea270 2024-07-26T06:15:31.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.109+0000 7f70c9400640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 0x7f70cc15d160 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f70b800e740 tx=0x7f70b800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:31.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.110+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b800cc60 con 0x7f70cc1ea270 2024-07-26T06:15:31.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.110+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70b800ee50 con 0x7f70cc1ea270 2024-07-26T06:15:31.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.110+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b80116b0 con 0x7f70cc1ea270 2024-07-26T06:15:31.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.110+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70cc15db40 con 0x7f70cc1ea270 2024-07-26T06:15:31.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.110+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70cc15e100 con 0x7f70cc1ea270 2024-07-26T06:15:31.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.111+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70cc0b4380 con 0x7f70cc1ea270 2024-07-26T06:15:31.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.112+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f70b8011850 con 0x7f70cc1ea270 2024-07-26T06:15:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.113+0000 7f70c2000640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f70ac076510 0x7f70ac0789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.113+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f70b8098800 con 0x7f70cc1ea270 2024-07-26T06:15:31.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.114+0000 7f70c9e00640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f70ac076510 0x7f70ac0789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:31.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.115+0000 7f70c9e00640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f70ac076510 0x7f70ac0789d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f70bc032000 tx=0x7f70bc001510 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:31.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.117+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70b8016030 con 0x7f70cc1ea270 2024-07-26T06:15:31.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.361+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70cc154f10 con 0x7f70cc1ea270 2024-07-26T06:15:31.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.362+0000 7f70c2000640 1 -- 172.21.15.163:0/2760627251 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f70b8061b80 con 0x7f70cc1ea270 2024-07-26T06:15:31.363 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:31.363 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f70ac076510 msgr2=0x7f70ac0789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f70ac076510 0x7f70ac0789d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f70bc032000 tx=0x7f70bc001510 comp rx=0 tx=0).stop 2024-07-26T06:15:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 msgr2=0x7f70cc15d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 0x7f70cc15d160 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f70b800e740 tx=0x7f70b800ec00 comp rx=0 tx=0).stop 2024-07-26T06:15:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 shutdown_connections 2024-07-26T06:15:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f70ac076510 0x7f70ac0789d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70cc15d6a0 0x7f70cc1e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70cc1ea270 0x7f70cc15d160 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 --2- 172.21.15.163:0/2760627251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b0003a80 0x7f70cc14fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 >> 172.21.15.163:0/2760627251 conn(0x7f70cc0b0e30 msgr2=0x7f70cc0b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 shutdown_connections 2024-07-26T06:15:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:31.365+0000 7f70cb200640 1 -- 172.21.15.163:0/2760627251 wait complete. 2024-07-26T06:15:31.385 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:31.607 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":7.2318790570000004,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.636567664815392,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:31.607 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.63656766481539 }, "state": "open", "uptime": 7.231879057 } ] 2024-07-26T06:15:31.608 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:15:36.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:36.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 -- 172.21.15.163:0/521405015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f4071a70 msgr2=0x7f07f40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:36.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 --2- 172.21.15.163:0/521405015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f4071a70 0x7f07f40b3710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f07f406cf00 tx=0x7f07ec02aa30 comp rx=0 tx=0).stop 2024-07-26T06:15:36.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 -- 172.21.15.163:0/521405015 shutdown_connections 2024-07-26T06:15:36.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 --2- 172.21.15.163:0/521405015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f4071a70 0x7f07f40b3710 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:36.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 --2- 172.21.15.163:0/521405015 >> v2:172.21.15.164:3300/0 conn(0x7f07f40ce4a0 0x7f07f4071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:36.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 -- 172.21.15.163:0/521405015 >> 172.21.15.163:0/521405015 conn(0x7f07f40b0e30 msgr2=0x7f07f40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:36.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 -- 172.21.15.163:0/521405015 shutdown_connections 2024-07-26T06:15:36.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.822+0000 7f07f9a00640 1 -- 172.21.15.163:0/521405015 wait complete. 2024-07-26T06:15:36.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.824+0000 7f07f9a00640 1 Processor -- start 2024-07-26T06:15:36.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.824+0000 7f07f9a00640 1 -- start start 2024-07-26T06:15:36.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07f4071a70 0x7f07f414cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 0x7f07f4148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07f41556f0 0x7f07f4153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07f40b3fe0 con 0x7f07f40ce4a0 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07f40b3e60 con 0x7f07f41556f0 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f07f40b4160 con 0x7f07f4071a70 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 0x7f07f4148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 0x7f07f4148b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33908/0 (socket says 172.21.15.163:33908) 2024-07-26T06:15:36.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 -- 172.21.15.163:0/3859302815 learned_addr learned my addr 172.21.15.163:0/3859302815 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:36.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f3e00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07f4071a70 0x7f07f414cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 -- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07f4071a70 msgr2=0x7f07f414cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f9000640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07f41556f0 0x7f07f4153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07f4071a70 0x7f07f414cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 -- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07f41556f0 msgr2=0x7f07f4153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07f41556f0 0x7f07f4153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07ebe00640 1 -- 172.21.15.163:0/3859302815 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f414d5b0 con 0x7f07f40ce4a0 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.825+0000 7f07f3e00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07f4071a70 0x7f07f414cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.826+0000 7f07ebe00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 0x7f07f4148b00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f07e400d770 tx=0x7f07e400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.826+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4017070 con 0x7f07f40ce4a0 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.826+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07e4009d20 con 0x7f07f40ce4a0 2024-07-26T06:15:36.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.826+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4005280 con 0x7f07f40ce4a0 2024-07-26T06:15:36.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.826+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f414d890 con 0x7f07f40ce4a0 2024-07-26T06:15:36.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.826+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f414de50 con 0x7f07f40ce4a0 2024-07-26T06:15:36.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.827+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f40b46f0 con 0x7f07f40ce4a0 2024-07-26T06:15:36.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.829+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f07e4005420 con 0x7f07f40ce4a0 2024-07-26T06:15:36.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.830+0000 7f07f0c00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f07d0076510 0x7f07d00789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:36.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.830+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f07e4097e20 con 0x7f07f40ce4a0 2024-07-26T06:15:36.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.831+0000 7f07f3e00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f07d0076510 0x7f07d00789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:36.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.832+0000 7f07f3e00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f07d0076510 0x7f07d00789d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00a810 tx=0x7f07dc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:36.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:36.835+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f07e40612d0 con 0x7f07f40ce4a0 2024-07-26T06:15:37.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.088+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07f40ced60 con 0x7f07f40ce4a0 2024-07-26T06:15:37.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.090+0000 7f07f0c00640 1 -- 172.21.15.163:0/3859302815 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f07e40692c0 con 0x7f07f40ce4a0 2024-07-26T06:15:37.091 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:37.091 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:37.091 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:37.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f07d0076510 msgr2=0x7f07d00789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:37.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f07d0076510 0x7f07d00789d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00a810 tx=0x7f07dc008040 comp rx=0 tx=0).stop 2024-07-26T06:15:37.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 msgr2=0x7f07f4148b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:37.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 0x7f07f4148b00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f07e400d770 tx=0x7f07e400dc30 comp rx=0 tx=0).stop 2024-07-26T06:15:37.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 shutdown_connections 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f07d0076510 0x7f07d00789d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07f41556f0 0x7f07f4153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07f40ce4a0 0x7f07f4148b00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 --2- 172.21.15.163:0/3859302815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07f4071a70 0x7f07f414cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 >> 172.21.15.163:0/3859302815 conn(0x7f07f40b0e30 msgr2=0x7f07f40b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 shutdown_connections 2024-07-26T06:15:37.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:37.093+0000 7f07f9a00640 1 -- 172.21.15.163:0/3859302815 wait complete. 2024-07-26T06:15:37.114 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:37.348 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":12.971783063,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.351517970191473,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:37.348 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.351517970191473 }, "state": "open", "uptime": 12.971783063 } ] 2024-07-26T06:15:37.348 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:15:42.349 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:42.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.564+0000 7fb43be00640 1 -- 172.21.15.163:0/2221437644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cd0c0 msgr2=0x7fb43c15a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:42.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.564+0000 7fb43be00640 1 --2- 172.21.15.163:0/2221437644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cd0c0 0x7fb43c15a6b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb434005fd0 tx=0x7fb43402a8b0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.564+0000 7fb43be00640 1 -- 172.21.15.163:0/2221437644 shutdown_connections 2024-07-26T06:15:42.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.564+0000 7fb43be00640 1 --2- 172.21.15.163:0/2221437644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cd0c0 0x7fb43c15a6b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.564+0000 7fb43be00640 1 -- 172.21.15.163:0/2221437644 >> 172.21.15.163:0/2221437644 conn(0x7fb43c0b0e30 msgr2=0x7fb43c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:42.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.565+0000 7fb43be00640 1 -- 172.21.15.163:0/2221437644 shutdown_connections 2024-07-26T06:15:42.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.565+0000 7fb43be00640 1 -- 172.21.15.163:0/2221437644 wait complete. 2024-07-26T06:15:42.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.567+0000 7fb43be00640 1 Processor -- start 2024-07-26T06:15:42.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.567+0000 7fb43be00640 1 -- start start 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb43be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb43c0741f0 0x7fb43c1f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb43be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 0x7fb43c1f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb43be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb43c0cd0c0 0x7fb43c0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb43be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c0b3560 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb43be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c0b33e0 con 0x7fb43c0cd0c0 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb43be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb43c0b36e0 con 0x7fb43c0741f0 2024-07-26T06:15:42.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb433e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 0x7fb43c1f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:42.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb433e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 0x7fb43c1f7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33928/0 (socket says 172.21.15.163:33928) 2024-07-26T06:15:42.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.568+0000 7fb433e00640 1 -- 172.21.15.163:0/2898255705 learned_addr learned my addr 172.21.15.163:0/2898255705 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb43b400640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb43c0cd0c0 0x7fb43c0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb433e00640 1 -- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb43c0741f0 msgr2=0x7fb43c1f57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb433e00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb43c0741f0 0x7fb43c1f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb433e00640 1 -- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb43c0cd0c0 msgr2=0x7fb43c0ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb433e00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb43c0cd0c0 0x7fb43c0ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb433e00640 1 -- 172.21.15.163:0/2898255705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c1e6ab0 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb433e00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 0x7fb43c1f7490 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c430 tx=0x7fb42400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424006890 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb424006b40 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424007020 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.569+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c1e6d90 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.570+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c1582e0 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.571+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43c0b3c30 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.572+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7fb424019020 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.573+0000 7fb432a00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb418076510 0x7fb4180789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:42.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.573+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fb4240972a0 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.573+0000 7fb43aa00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb418076510 0x7fb4180789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:42.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.575+0000 7fb43aa00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb418076510 0x7fb4180789d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb42c006fd0 tx=0x7fb42c006bc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:42.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.577+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb424060920 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.820+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb43c0b4aa0 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.822+0000 7fb432a00640 1 -- 172.21.15.163:0/2898255705 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7fb424060220 con 0x7fb43c0cc1c0 2024-07-26T06:15:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:42.823 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:42.823 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.824+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb418076510 msgr2=0x7fb4180789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.824+0000 7fb43be00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb418076510 0x7fb4180789d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb42c006fd0 tx=0x7fb42c006bc0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.824+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 msgr2=0x7fb43c1f7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.824+0000 7fb43be00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 0x7fb43c1f7490 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c430 tx=0x7fb42400c8f0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 shutdown_connections 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb418076510 0x7fb4180789d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb43c0cd0c0 0x7fb43c0ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb43c0cc1c0 0x7fb43c1f7490 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 --2- 172.21.15.163:0/2898255705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb43c0741f0 0x7fb43c1f57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 >> 172.21.15.163:0/2898255705 conn(0x7fb43c0b0e30 msgr2=0x7fb43c0722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 shutdown_connections 2024-07-26T06:15:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:42.825+0000 7fb43be00640 1 -- 172.21.15.163:0/2898255705 wait complete. 2024-07-26T06:15:42.845 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":18.698687284999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.07085651827677,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:43.074 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.07085651827677 }, "state": "open", "uptime": 18.698687285 } ] 2024-07-26T06:15:43.074 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:15:48.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:48.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 -- 172.21.15.163:0/1530570249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 msgr2=0x7fceec0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 --2- 172.21.15.163:0/1530570249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec0b3710 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fceec06cf00 tx=0x7fcee402a8b0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 -- 172.21.15.163:0/1530570249 shutdown_connections 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 --2- 172.21.15.163:0/1530570249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec0b3710 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 --2- 172.21.15.163:0/1530570249 >> v2:172.21.15.164:3300/0 conn(0x7fceec0ce4a0 0x7fceec071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 -- 172.21.15.163:0/1530570249 >> 172.21.15.163:0/1530570249 conn(0x7fceec0b0e30 msgr2=0x7fceec0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 -- 172.21.15.163:0/1530570249 shutdown_connections 2024-07-26T06:15:48.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.284+0000 7fcef0a00640 1 -- 172.21.15.163:0/1530570249 wait complete. 2024-07-26T06:15:48.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.286+0000 7fcef0a00640 1 Processor -- start 2024-07-26T06:15:48.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.286+0000 7fcef0a00640 1 -- start start 2024-07-26T06:15:48.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fcef0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:48.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fcef0a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fceec0ce4a0 0x7fceec0aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:48.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fcef0a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fceec1556f0 0x7fceec0b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:48.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fcef0a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fceec0b3fe0 con 0x7fceec071a70 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fcef0a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fceec0b3e60 con 0x7fceec1556f0 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fcef0a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fceec0b4160 con 0x7fceec0ce4a0 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fceeb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fceeb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33444/0 (socket says 172.21.15.163:33444) 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.287+0000 7fceeb400640 1 -- 172.21.15.163:0/3493025152 learned_addr learned my addr 172.21.15.163:0/3493025152 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fceeb400640 1 -- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fceec0ce4a0 msgr2=0x7fceec0aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fceeb400640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fceec0ce4a0 0x7fceec0aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fceeb400640 1 -- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fceec1556f0 msgr2=0x7fceec0b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fceeb400640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fceec1556f0 0x7fceec0b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fceeb400640 1 -- 172.21.15.163:0/3493025152 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceec0ad840 con 0x7fceec071a70 2024-07-26T06:15:48.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fceeb400640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec150b60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00c430 tx=0x7fcedc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:48.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc011790 con 0x7fceec071a70 2024-07-26T06:15:48.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcedc017070 con 0x7fceec071a70 2024-07-26T06:15:48.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc011ce0 con 0x7fceec071a70 2024-07-26T06:15:48.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.288+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceec0adb20 con 0x7fceec071a70 2024-07-26T06:15:48.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.289+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceec0ae0e0 con 0x7fceec071a70 2024-07-26T06:15:48.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.290+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceec0ae410 con 0x7fceec071a70 2024-07-26T06:15:48.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.291+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7fcedc01c460 con 0x7fceec071a70 2024-07-26T06:15:48.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.292+0000 7fcee3400640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcec8076510 0x7fcec80789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:48.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.292+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fcedc097a00 con 0x7fceec071a70 2024-07-26T06:15:48.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.292+0000 7fceeaa00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcec8076510 0x7fcec80789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:48.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.293+0000 7fceeaa00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcec8076510 0x7fcec80789d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fceec0cbaf0 tx=0x7fced4005c70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:48.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.296+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcedc060ff0 con 0x7fceec071a70 2024-07-26T06:15:48.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.538+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fceec0b46f0 con 0x7fceec071a70 2024-07-26T06:15:48.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.539+0000 7fcee3400640 1 -- 172.21.15.163:0/3493025152 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7fcedc0608f0 con 0x7fceec071a70 2024-07-26T06:15:48.540 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:48.540 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:48.540 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcec8076510 msgr2=0x7fcec80789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcec8076510 0x7fcec80789d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fceec0cbaf0 tx=0x7fced4005c70 comp rx=0 tx=0).stop 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 msgr2=0x7fceec150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec150b60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00c430 tx=0x7fcedc00c8f0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 shutdown_connections 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcec8076510 0x7fcec80789d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fceec1556f0 0x7fceec0b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fceec0ce4a0 0x7fceec0aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 --2- 172.21.15.163:0/3493025152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fceec071a70 0x7fceec150b60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:48.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 >> 172.21.15.163:0/3493025152 conn(0x7fceec0b0e30 msgr2=0x7fceec15d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:48.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 shutdown_connections 2024-07-26T06:15:48.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:48.542+0000 7fcef0a00640 1 -- 172.21.15.163:0/3493025152 wait complete. 2024-07-26T06:15:48.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:48.790 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":11,"uptime":24.415591675999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.794364743431686,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:48.865 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.794364743431686 }, "state": "open", "uptime": 24.415591676 } ] 2024-07-26T06:15:48.866 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:15:53.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 -- 172.21.15.163:0/1850337540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f41556f0 msgr2=0x7f26f40cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 --2- 172.21.15.163:0/1850337540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f41556f0 0x7f26f40cdd90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f26dc005fd0 tx=0x7f26dc02a5d0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 -- 172.21.15.163:0/1850337540 shutdown_connections 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 --2- 172.21.15.163:0/1850337540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26f40ce2d0 0x7f26f4071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 --2- 172.21.15.163:0/1850337540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f41556f0 0x7f26f40cdd90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 -- 172.21.15.163:0/1850337540 >> 172.21.15.163:0/1850337540 conn(0x7f26f40b0e30 msgr2=0x7f26f40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 -- 172.21.15.163:0/1850337540 shutdown_connections 2024-07-26T06:15:54.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.083+0000 7f26f3200640 1 -- 172.21.15.163:0/1850337540 wait complete. 2024-07-26T06:15:54.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.085+0000 7f26f3200640 1 Processor -- start 2024-07-26T06:15:54.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.085+0000 7f26f3200640 1 -- start start 2024-07-26T06:15:54.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f3200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26f4071a70 0x7f26f41e8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:54.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f40ce2d0 0x7f26f41e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 0x7f26f415a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f3200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26f415f310 con 0x7f26f41556f0 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f3200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f26f4073660 con 0x7f26f40ce2d0 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f3200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f26f415f490 con 0x7f26f4071a70 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 0x7f26f415a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 0x7f26f415a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33448/0 (socket says 172.21.15.163:33448) 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f2800640 1 -- 172.21.15.163:0/3848350683 learned_addr learned my addr 172.21.15.163:0/3848350683 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f1e00640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26f4071a70 0x7f26f41e8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.086+0000 7f26f1400640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f40ce2d0 0x7f26f41e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f2800640 1 -- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26f4071a70 msgr2=0x7f26f41e8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f2800640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26f4071a70 0x7f26f41e8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f2800640 1 -- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f40ce2d0 msgr2=0x7f26f41e5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:54.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f2800640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f40ce2d0 0x7f26f41e5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f2800640 1 -- 172.21.15.163:0/3848350683 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f41e5f60 con 0x7f26f41556f0 2024-07-26T06:15:54.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f2800640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 0x7f26f415a2c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00c3c0 tx=0x7f26ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:54.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec017070 con 0x7f26f41556f0 2024-07-26T06:15:54.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26ec0053c0 con 0x7f26f41556f0 2024-07-26T06:15:54.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec003850 con 0x7f26f41556f0 2024-07-26T06:15:54.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f41e6240 con 0x7f26f41556f0 2024-07-26T06:15:54.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.087+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f41e6800 con 0x7f26f41556f0 2024-07-26T06:15:54.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.089+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f4071530 con 0x7f26f41556f0 2024-07-26T06:15:54.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.090+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f26ec01a070 con 0x7f26f41556f0 2024-07-26T06:15:54.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.091+0000 7f26ea000640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26d0076510 0x7f26d00789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:54.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.091+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f26ec096f50 con 0x7f26f41556f0 2024-07-26T06:15:54.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.091+0000 7f26f1e00640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26d0076510 0x7f26d00789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:54.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.093+0000 7f26f1e00640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26d0076510 0x7f26d00789d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f26dc033000 tx=0x7f26dc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:54.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.095+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26ec060540 con 0x7f26f41556f0 2024-07-26T06:15:54.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.339+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26f41f4500 con 0x7f26f41556f0 2024-07-26T06:15:54.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.340+0000 7f26ea000640 1 -- 172.21.15.163:0/3848350683 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f26ec05fe40 con 0x7f26f41556f0 2024-07-26T06:15:54.341 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:15:54.342 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:15:54.342 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:15:54.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26d0076510 msgr2=0x7f26d00789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26d0076510 0x7f26d00789d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f26dc033000 tx=0x7f26dc004730 comp rx=0 tx=0).stop 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 msgr2=0x7f26f415a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 0x7f26f415a2c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00c3c0 tx=0x7f26ec00c880 comp rx=0 tx=0).stop 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 shutdown_connections 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f26d0076510 0x7f26d00789d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26f41556f0 0x7f26f415a2c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26f40ce2d0 0x7f26f41e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 --2- 172.21.15.163:0/3848350683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26f4071a70 0x7f26f41e8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.343+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 >> 172.21.15.163:0/3848350683 conn(0x7f26f40b0e30 msgr2=0x7f26f40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.344+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 shutdown_connections 2024-07-26T06:15:54.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:54.344+0000 7f26f3200640 1 -- 172.21.15.163:0/3848350683 wait complete. 2024-07-26T06:15:54.365 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:15:54.603 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":30.228494459,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.516949859999951,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:15:54.603 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.51694985999995 }, "state": "open", "uptime": 30.228494459 } ] 2024-07-26T06:15:54.604 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:15:56.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:15:56.341 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:15:59.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.822+0000 7fcadf000640 1 -- 172.21.15.163:0/1039461471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad815d5a0 msgr2=0x7fcad80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.822+0000 7fcadf000640 1 --2- 172.21.15.163:0/1039461471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad815d5a0 0x7fcad80ce2f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009830 tx=0x7fcac802a480 comp rx=0 tx=0).stop 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.822+0000 7fcadf000640 1 -- 172.21.15.163:0/1039461471 shutdown_connections 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.822+0000 7fcadf000640 1 --2- 172.21.15.163:0/1039461471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad815d5a0 0x7fcad80ce2f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.822+0000 7fcadf000640 1 -- 172.21.15.163:0/1039461471 >> 172.21.15.163:0/1039461471 conn(0x7fcad80cfda0 msgr2=0x7fcad80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.822+0000 7fcadf000640 1 -- 172.21.15.163:0/1039461471 shutdown_connections 2024-07-26T06:15:59.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.823+0000 7fcadf000640 1 -- 172.21.15.163:0/1039461471 wait complete. 2024-07-26T06:15:59.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.824+0000 7fcadf000640 1 Processor -- start 2024-07-26T06:15:59.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.824+0000 7fcadf000640 1 -- start start 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.825+0000 7fcadf000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcad80b1e80 0x7fcad814c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.825+0000 7fcadf000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 0x7fcad80c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.825+0000 7fcadf000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcad815d5a0 0x7fcad80cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.825+0000 7fcadf000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcad80b6f60 con 0x7fcad80ce830 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.825+0000 7fcadf000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcad80b6de0 con 0x7fcad815d5a0 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.825+0000 7fcadf000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcad80b54b0 con 0x7fcad80b1e80 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 0x7fcad80c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:59.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 0x7fcad80c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33110/0 (socket says 172.21.15.163:33110) 2024-07-26T06:15:59.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 -- 172.21.15.163:0/414609398 learned_addr learned my addr 172.21.15.163:0/414609398 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:15:59.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcaddc00640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcad80b1e80 0x7fcad814c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:59.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcade600640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcad815d5a0 0x7fcad80cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:59.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 -- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcad80b1e80 msgr2=0x7fcad814c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcad80b1e80 0x7fcad814c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 -- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcad815d5a0 msgr2=0x7fcad80cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcad815d5a0 0x7fcad80cc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.826+0000 7fcadd200640 1 -- 172.21.15.163:0/414609398 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad81f8240 con 0x7fcad80ce830 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.827+0000 7fcadd200640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 0x7fcad80c6e50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcac400c7f0 tx=0x7fcac400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.827+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4016020 con 0x7fcad80ce830 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.827+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcac400ab40 con 0x7fcad80ce830 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.827+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4012420 con 0x7fcad80ce830 2024-07-26T06:15:59.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.827+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad81f7f60 con 0x7fcad80ce830 2024-07-26T06:15:59.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.827+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad80ca7e0 con 0x7fcad80ce830 2024-07-26T06:15:59.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.828+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad80cf5a0 con 0x7fcad80ce830 2024-07-26T06:15:59.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.829+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7fcac4019020 con 0x7fcad80ce830 2024-07-26T06:15:59.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.830+0000 7fcad5600640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcab8076510 0x7fcab80789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:15:59.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.830+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fcac40979c0 con 0x7fcad80ce830 2024-07-26T06:15:59.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.831+0000 7fcaddc00640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcab8076510 0x7fcab80789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:15:59.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.832+0000 7fcaddc00640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcab8076510 0x7fcab80789d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcac8034000 tx=0x7fcac8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:15:59.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:15:59.835+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcad80cf5a0 con 0x7fcad80ce830 2024-07-26T06:16:00.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.082+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcad8068240 con 0x7fcad80ce830 2024-07-26T06:16:00.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.083+0000 7fcad5600640 1 -- 172.21.15.163:0/414609398 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7fcac4099e20 con 0x7fcad80ce830 2024-07-26T06:16:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:00.085 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:00.085 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:00.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcab8076510 msgr2=0x7fcab80789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:00.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcab8076510 0x7fcab80789d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcac8034000 tx=0x7fcac8004730 comp rx=0 tx=0).stop 2024-07-26T06:16:00.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 msgr2=0x7fcad80c6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:00.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 0x7fcad80c6e50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcac400c7f0 tx=0x7fcac400ccb0 comp rx=0 tx=0).stop 2024-07-26T06:16:00.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 shutdown_connections 2024-07-26T06:16:00.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fcab8076510 0x7fcab80789d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:00.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcad815d5a0 0x7fcad80cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:00.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcad80ce830 0x7fcad80c6e50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:00.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 --2- 172.21.15.163:0/414609398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcad80b1e80 0x7fcad814c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:00.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 >> 172.21.15.163:0/414609398 conn(0x7fcad80cfda0 msgr2=0x7fcad815b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:00.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 shutdown_connections 2024-07-26T06:16:00.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:00.086+0000 7fcadf000640 1 -- 172.21.15.163:0/414609398 wait complete. 2024-07-26T06:16:00.106 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:00.338 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":35.963398433000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.246884405957069,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:00.339 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.24688440595707 }, "state": "open", "uptime": 35.963398433 } ] 2024-07-26T06:16:00.339 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:05.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:05.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 -- 172.21.15.163:0/3158885450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca4071a70 msgr2=0x7f4ca40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 --2- 172.21.15.163:0/3158885450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca4071a70 0x7f4ca40b3710 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005fd0 tx=0x7f4c9c02aa90 comp rx=0 tx=0).stop 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 -- 172.21.15.163:0/3158885450 shutdown_connections 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 --2- 172.21.15.163:0/3158885450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca4071a70 0x7f4ca40b3710 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 --2- 172.21.15.163:0/3158885450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ca40ce4a0 0x7f4ca4071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 -- 172.21.15.163:0/3158885450 >> 172.21.15.163:0/3158885450 conn(0x7f4ca40b0e30 msgr2=0x7f4ca40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 -- 172.21.15.163:0/3158885450 shutdown_connections 2024-07-26T06:16:05.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.554+0000 7f4ca3200640 1 -- 172.21.15.163:0/3158885450 wait complete. 2024-07-26T06:16:05.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.556+0000 7f4ca3200640 1 Processor -- start 2024-07-26T06:16:05.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.556+0000 7f4ca3200640 1 -- start start 2024-07-26T06:16:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca3200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ca4071a70 0x7f4ca41375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ca40ce4a0 0x7f4ca4151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 0x7f4ca4133fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca3200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca40b4020 con 0x7f4ca41556f0 2024-07-26T06:16:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca3200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca40b3ea0 con 0x7f4ca40ce4a0 2024-07-26T06:16:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca3200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca40b41a0 con 0x7f4ca4071a70 2024-07-26T06:16:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 0x7f4ca4133fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 0x7f4ca4133fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49210/0 (socket says 172.21.15.163:49210) 2024-07-26T06:16:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.557+0000 7f4ca2800640 1 -- 172.21.15.163:0/1912060795 learned_addr learned my addr 172.21.15.163:0/1912060795 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca1e00640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ca4071a70 0x7f4ca41375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca2800640 1 -- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ca4071a70 msgr2=0x7f4ca41375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca2800640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ca4071a70 0x7f4ca41375c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca2800640 1 -- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ca40ce4a0 msgr2=0x7f4ca4151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4c99400640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ca40ce4a0 0x7f4ca4151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca2800640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ca40ce4a0 0x7f4ca4151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca2800640 1 -- 172.21.15.163:0/1912060795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca4152350 con 0x7f4ca41556f0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4ca2800640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 0x7f4ca4133fd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c006010 tx=0x7f4c9c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.558+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c046070 con 0x7f4ca41556f0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.559+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c9c003e50 con 0x7f4ca41556f0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.559+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0044b0 con 0x7f4ca41556f0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.559+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca4152630 con 0x7f4ca41556f0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.559+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca41542f0 con 0x7f4ca41556f0 2024-07-26T06:16:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.560+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca40b4730 con 0x7f4ca41556f0 2024-07-26T06:16:05.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.561+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f4c9c049070 con 0x7f4ca41556f0 2024-07-26T06:16:05.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.562+0000 7f4c9aa00640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4c80076510 0x7f4c800789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:05.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.563+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f4c9c0b8c40 con 0x7f4ca41556f0 2024-07-26T06:16:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.563+0000 7f4ca1e00640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4c80076510 0x7f4c800789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:05.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.564+0000 7f4ca1e00640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4c80076510 0x7f4c800789d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4c940015a0 tx=0x7f4c94009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:05.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.567+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c9c032030 con 0x7f4ca41556f0 2024-07-26T06:16:05.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.805+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ca40b0bb0 con 0x7f4ca41556f0 2024-07-26T06:16:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.806+0000 7f4c9aa00640 1 -- 172.21.15.163:0/1912060795 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f4c9c082050 con 0x7f4ca41556f0 2024-07-26T06:16:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:05.807 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4c80076510 msgr2=0x7f4c800789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4c80076510 0x7f4c800789d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4c940015a0 tx=0x7f4c94009040 comp rx=0 tx=0).stop 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 msgr2=0x7f4ca4133fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 0x7f4ca4133fd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c006010 tx=0x7f4c9c003ce0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 shutdown_connections 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4c80076510 0x7f4c800789d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4ca41556f0 0x7f4ca4133fd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ca40ce4a0 0x7f4ca4151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 --2- 172.21.15.163:0/1912060795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ca4071a70 0x7f4ca41375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:05.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 >> 172.21.15.163:0/1912060795 conn(0x7f4ca40b0e30 msgr2=0x7f4ca40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:05.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 shutdown_connections 2024-07-26T06:16:05.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:05.809+0000 7f4ca3200640 1 -- 172.21.15.163:0/1912060795 wait complete. 2024-07-26T06:16:05.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:06.066 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":41.691302499000003,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.980696984210475,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:06.067 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.980696984210475 }, "state": "open", "uptime": 41.691302499 } ] 2024-07-26T06:16:06.067 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:11.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:11.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 -- 172.21.15.163:0/408697233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30e8003a00 msgr2=0x7f30e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 --2- 172.21.15.163:0/408697233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30e8003a00 0x7f30e8003e00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f30f400bb20 tx=0x7f30e8005830 comp rx=0 tx=0).stop 2024-07-26T06:16:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 -- 172.21.15.163:0/408697233 shutdown_connections 2024-07-26T06:16:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 --2- 172.21.15.163:0/408697233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30e8003a00 0x7f30e8003e00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 -- 172.21.15.163:0/408697233 >> 172.21.15.163:0/408697233 conn(0x7f310406d420 msgr2=0x7f310406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 -- 172.21.15.163:0/408697233 shutdown_connections 2024-07-26T06:16:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.285+0000 7f3109a00640 1 -- 172.21.15.163:0/408697233 wait complete. 2024-07-26T06:16:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.287+0000 7f3109a00640 1 Processor -- start 2024-07-26T06:16:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.287+0000 7f3109a00640 1 -- start start 2024-07-26T06:16:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3109a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30e8003a00 0x7f310414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3109a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f310414ad70 0x7f310414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3109a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 0x7f31041f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3109a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f310406c320 con 0x7f31041f6ef0 2024-07-26T06:16:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3109a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f310415f040 con 0x7f30e8003a00 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3109a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f310406c4a0 con 0x7f310414ad70 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3103400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 0x7f31041f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3103400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 0x7f31041f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:49224/0 (socket says 172.21.15.163:49224) 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.288+0000 7f3103400640 1 -- 172.21.15.163:0/3480503347 learned_addr learned my addr 172.21.15.163:0/3480503347 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3102000640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f310414ad70 0x7f310414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3102a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30e8003a00 0x7f310414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3103400640 1 -- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f310414ad70 msgr2=0x7f310414a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3103400640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f310414ad70 0x7f310414a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3103400640 1 -- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30e8003a00 msgr2=0x7f310414a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3103400640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30e8003a00 0x7f310414a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3103400640 1 -- 172.21.15.163:0/3480503347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31040cea20 con 0x7f31041f6ef0 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3103400640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 0x7f31041f7370 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f31040b4210 tx=0x7f30fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0048d0 con 0x7f31041f6ef0 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30fc004a90 con 0x7f31041f6ef0 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310414c120 con 0x7f31041f6ef0 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.289+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc005030 con 0x7f31041f6ef0 2024-07-26T06:16:11.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.290+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31041f78b0 con 0x7f31041f6ef0 2024-07-26T06:16:11.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.291+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310415f1c0 con 0x7f31041f6ef0 2024-07-26T06:16:11.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.292+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f30fc0051d0 con 0x7f31041f6ef0 2024-07-26T06:16:11.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.293+0000 7f30faa00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30e4076510 0x7f30e40789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:11.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.294+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f30fc097680 con 0x7f31041f6ef0 2024-07-26T06:16:11.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.294+0000 7f3102a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30e4076510 0x7f30e40789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:11.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.295+0000 7f3102a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30e4076510 0x7f30e40789d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f30f4036000 tx=0x7f30f40046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:11.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.298+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f30fc060cd0 con 0x7f31041f6ef0 2024-07-26T06:16:11.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.546+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3104069050 con 0x7f31041f6ef0 2024-07-26T06:16:11.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.546+0000 7f30faa00640 1 -- 172.21.15.163:0/3480503347 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f30fc0605d0 con 0x7f31041f6ef0 2024-07-26T06:16:11.547 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:11.547 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:11.548 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:11.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.549+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30e4076510 msgr2=0x7f30e40789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:11.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.549+0000 7f3109a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30e4076510 0x7f30e40789d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f30f4036000 tx=0x7f30f40046c0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.549+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 msgr2=0x7f31041f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:11.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.549+0000 7f3109a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 0x7f31041f7370 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f31040b4210 tx=0x7f30fc00dc10 comp rx=0 tx=0).stop 2024-07-26T06:16:11.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 shutdown_connections 2024-07-26T06:16:11.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30e4076510 0x7f30e40789d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31041f6ef0 0x7f31041f7370 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f310414ad70 0x7f310414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 --2- 172.21.15.163:0/3480503347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30e8003a00 0x7f310414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:11.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 >> 172.21.15.163:0/3480503347 conn(0x7f310406d420 msgr2=0x7f31040b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:11.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 shutdown_connections 2024-07-26T06:16:11.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:11.550+0000 7f3109a00640 1 -- 172.21.15.163:0/3480503347 wait complete. 2024-07-26T06:16:11.569 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:11.799 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":47.424206480000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.717781363773724,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:11.800 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.717781363773724 }, "state": "open", "uptime": 47.42420648 } ] 2024-07-26T06:16:11.800 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:16.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae9200640 1 -- 172.21.15.163:0/3673973788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ac4003a00 msgr2=0x7f1ac4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae9200640 1 --2- 172.21.15.163:0/3673973788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ac4003a00 0x7f1ac4003e20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009b50 tx=0x7f1ac4005830 comp rx=0 tx=0).stop 2024-07-26T06:16:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae2800640 1 -- 172.21.15.163:0/3673973788 reap_dead start 2024-07-26T06:16:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae9200640 1 -- 172.21.15.163:0/3673973788 shutdown_connections 2024-07-26T06:16:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae9200640 1 -- 172.21.15.163:0/3673973788 >> 172.21.15.163:0/3673973788 conn(0x7f1ae40b2f80 msgr2=0x7f1ae40b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae9200640 1 -- 172.21.15.163:0/3673973788 shutdown_connections 2024-07-26T06:16:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.019+0000 7f1ae9200640 1 -- 172.21.15.163:0/3673973788 wait complete. 2024-07-26T06:16:17.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.021+0000 7f1ae9200640 1 Processor -- start 2024-07-26T06:16:17.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.021+0000 7f1ae9200640 1 -- start start 2024-07-26T06:16:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae9200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ac4003a00 0x7f1ae415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae9200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 0x7f1ae40ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae9200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ae406e8c0 0x7f1ae40aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae9200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae40cfb80 con 0x7f1ae406e440 2024-07-26T06:16:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae9200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae40cfa00 con 0x7f1ae406e8c0 2024-07-26T06:16:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae9200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae40cfd00 con 0x7f1ac4003a00 2024-07-26T06:16:17.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 0x7f1ae40ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:17.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 0x7f1ae40ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36808/0 (socket says 172.21.15.163:36808) 2024-07-26T06:16:17.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.022+0000 7f1ae1400640 1 -- 172.21.15.163:0/2735907960 learned_addr learned my addr 172.21.15.163:0/2735907960 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:17.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae1400640 1 -- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ac4003a00 msgr2=0x7f1ae415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:17.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae2800640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ae406e8c0 0x7f1ae40aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:17.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae1400640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ac4003a00 0x7f1ae415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:17.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae1400640 1 -- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ae406e8c0 msgr2=0x7f1ae40aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:17.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae1400640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ae406e8c0 0x7f1ae40aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:17.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae1400640 1 -- 172.21.15.163:0/2735907960 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae4155900 con 0x7f1ae406e440 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae1400640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 0x7f1ae40ae930 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00cbc0 tx=0x7f1acc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc017070 con 0x7f1ae406e440 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1acc00ae30 con 0x7f1ae406e440 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc003bb0 con 0x7f1ae406e440 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.023+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae4153730 con 0x7f1ae406e440 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.024+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae40af780 con 0x7f1ae406e440 2024-07-26T06:16:17.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.025+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae40b0f70 con 0x7f1ae406e440 2024-07-26T06:16:17.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.026+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f1acc01a070 con 0x7f1ae406e440 2024-07-26T06:16:17.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.027+0000 7f1ada000640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1ac0076510 0x7f1ac00789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:17.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.028+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f1acc097670 con 0x7f1ae406e440 2024-07-26T06:16:17.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.028+0000 7f1ae1e00640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1ac0076510 0x7f1ac00789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:17.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.029+0000 7f1ae1e00640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1ac0076510 0x7f1ac00789d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4035000 tx=0x7f1ad4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:17.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.032+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1acc060b90 con 0x7f1ae406e440 2024-07-26T06:16:17.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.278+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ae4071690 con 0x7f1ae406e440 2024-07-26T06:16:17.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.279+0000 7f1ada000640 1 -- 172.21.15.163:0/2735907960 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f1acc060490 con 0x7f1ae406e440 2024-07-26T06:16:17.279 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:17.280 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:17.280 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:17.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.281+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1ac0076510 msgr2=0x7f1ac00789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.281+0000 7f1ae9200640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1ac0076510 0x7f1ac00789d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4035000 tx=0x7f1ad4004730 comp rx=0 tx=0).stop 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.281+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 msgr2=0x7f1ae40ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.281+0000 7f1ae9200640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 0x7f1ae40ae930 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00cbc0 tx=0x7f1acc00a430 comp rx=0 tx=0).stop 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.281+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 shutdown_connections 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.281+0000 7f1ae9200640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1ac0076510 0x7f1ac00789d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.282+0000 7f1ae9200640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ae406e8c0 0x7f1ae40aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.282+0000 7f1ae9200640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ae406e440 0x7f1ae40ae930 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.282+0000 7f1ae9200640 1 --2- 172.21.15.163:0/2735907960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ac4003a00 0x7f1ae415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.282+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 >> 172.21.15.163:0/2735907960 conn(0x7f1ae40b2f80 msgr2=0x7f1ae40b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:17.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.282+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 shutdown_connections 2024-07-26T06:16:17.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:17.282+0000 7f1ae9200640 1 -- 172.21.15.163:0/2735907960 wait complete. 2024-07-26T06:16:17.303 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:17.539 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":53.162110378000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.458100527158457,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:17.540 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.458100527158457 }, "state": "open", "uptime": 53.162110378 } ] 2024-07-26T06:16:17.540 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:22.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:22.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 -- 172.21.15.163:0/4242439223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac0cbcb0 msgr2=0x7f27ac0cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:22.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 --2- 172.21.15.163:0/4242439223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac0cbcb0 0x7f27ac0cc140 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f2798009830 tx=0x7f279802a510 comp rx=0 tx=0).stop 2024-07-26T06:16:22.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 -- 172.21.15.163:0/4242439223 shutdown_connections 2024-07-26T06:16:22.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 --2- 172.21.15.163:0/4242439223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac0cbcb0 0x7f27ac0cc140 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:22.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 -- 172.21.15.163:0/4242439223 >> 172.21.15.163:0/4242439223 conn(0x7f27ac06d420 msgr2=0x7f27ac06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:22.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 -- 172.21.15.163:0/4242439223 shutdown_connections 2024-07-26T06:16:22.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.756+0000 7f27b3e00640 1 -- 172.21.15.163:0/4242439223 wait complete. 2024-07-26T06:16:22.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.758+0000 7f27b3e00640 1 Processor -- start 2024-07-26T06:16:22.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.758+0000 7f27b3e00640 1 -- start start 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.759+0000 7f27b3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27ac0cbcb0 0x7f27ac0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.759+0000 7f27b3e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27ac1507d0 0x7f27ac0b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.759+0000 7f27b3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 0x7f27ac143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.759+0000 7f27b3e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f27ac15bf60 con 0x7f27ac15b240 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.759+0000 7f27b3e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f27ac15bde0 con 0x7f27ac1507d0 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.759+0000 7f27b3e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f27ac15c0e0 con 0x7f27ac0cbcb0 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 0x7f27ac143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 0x7f27ac143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:36836/0 (socket says 172.21.15.163:36836) 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 -- 172.21.15.163:0/997724823 learned_addr learned my addr 172.21.15.163:0/997724823 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1400640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27ac0cbcb0 0x7f27ac0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b0a00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27ac1507d0 0x7f27ac0b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 -- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27ac0cbcb0 msgr2=0x7f27ac0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27ac0cbcb0 0x7f27ac0b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 -- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27ac1507d0 msgr2=0x7f27ac0b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27ac1507d0 0x7f27ac0b1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b1e00640 1 -- 172.21.15.163:0/997724823 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27ac155700 con 0x7f27ac15b240 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.760+0000 7f27b0a00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27ac1507d0 0x7f27ac0b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.761+0000 7f27b1e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 0x7f27ac143270 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f27a400c6f0 tx=0x7f27a400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.761+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4017070 con 0x7f27ac15b240 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.761+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27a4003c90 con 0x7f27ac15b240 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.761+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27ac1559e0 con 0x7f27ac15b240 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.761+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4003e60 con 0x7f27ac15b240 2024-07-26T06:16:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.761+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27ac155fa0 con 0x7f27ac15b240 2024-07-26T06:16:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.762+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27ac0cc140 con 0x7f27ac15b240 2024-07-26T06:16:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.763+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f27a401a070 con 0x7f27ac15b240 2024-07-26T06:16:22.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.764+0000 7f27a9600640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f278c076510 0x7f278c0789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:22.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.764+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f27a4097220 con 0x7f27ac15b240 2024-07-26T06:16:22.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.765+0000 7f27b1400640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f278c076510 0x7f278c0789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:22.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.766+0000 7f27b1400640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f278c076510 0x7f278c0789d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27a0003d90 tx=0x7f27a0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:22.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:22.769+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f27a40607d0 con 0x7f27ac15b240 2024-07-26T06:16:23.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.015+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27ac152ff0 con 0x7f27ac15b240 2024-07-26T06:16:23.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.017+0000 7f27a9600640 1 -- 172.21.15.163:0/997724823 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f27a40600d0 con 0x7f27ac15b240 2024-07-26T06:16:23.018 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:23.018 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:23.018 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:23.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.019+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f278c076510 msgr2=0x7f278c0789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:23.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.019+0000 7f27b3e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f278c076510 0x7f278c0789d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27a0003d90 tx=0x7f27a0009040 comp rx=0 tx=0).stop 2024-07-26T06:16:23.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.019+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 msgr2=0x7f27ac143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:23.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 0x7f27ac143270 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f27a400c6f0 tx=0x7f27a400cbb0 comp rx=0 tx=0).stop 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 shutdown_connections 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f278c076510 0x7f278c0789d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27ac15b240 0x7f27ac143270 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27ac1507d0 0x7f27ac0b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 --2- 172.21.15.163:0/997724823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27ac0cbcb0 0x7f27ac0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 >> 172.21.15.163:0/997724823 conn(0x7f27ac06d420 msgr2=0x7f27ac14f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 shutdown_connections 2024-07-26T06:16:23.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:23.020+0000 7f27b3e00640 1 -- 172.21.15.163:0/997724823 wait complete. 2024-07-26T06:16:23.042 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:23.275 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":58.899014291,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.201884021347762,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:23.275 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.201884021347762 }, "state": "open", "uptime": 58.899014291 } ] 2024-07-26T06:16:23.276 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:26.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:16:26.471 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:16:28.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 -- 172.21.15.163:0/1898670778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780cdc40 msgr2=0x7f64780b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 --2- 172.21.15.163:0/1898670778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780cdc40 0x7f64780b2840 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6470009830 tx=0x7f647002a5c0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 -- 172.21.15.163:0/1898670778 shutdown_connections 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 --2- 172.21.15.163:0/1898670778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780cdc40 0x7f64780b2840 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 -- 172.21.15.163:0/1898670778 >> 172.21.15.163:0/1898670778 conn(0x7f64780b0e30 msgr2=0x7f64780b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 -- 172.21.15.163:0/1898670778 shutdown_connections 2024-07-26T06:16:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.491+0000 7f647dc00640 1 -- 172.21.15.163:0/1898670778 wait complete. 2024-07-26T06:16:28.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.493+0000 7f647dc00640 1 Processor -- start 2024-07-26T06:16:28.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.493+0000 7f647dc00640 1 -- start start 2024-07-26T06:16:28.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f647dc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 0x7f64780aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f647dc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64780cdc40 0x7f6478152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f647dc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64780aa930 0x7f6478152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f647dc00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64780b2d80 con 0x7f64780b4420 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f647dc00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f647806f500 con 0x7f64780cdc40 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f647dc00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f64780b2f00 con 0x7f64780aa930 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f6477e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 0x7f64780aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f6477e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 0x7f64780aa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52384/0 (socket says 172.21.15.163:52384) 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f6477e00640 1 -- 172.21.15.163:0/622414710 learned_addr learned my addr 172.21.15.163:0/622414710 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.494+0000 7f6477400640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64780cdc40 0x7f6478152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f647d200640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64780aa930 0x7f6478152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f6477e00640 1 -- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64780aa930 msgr2=0x7f6478152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f6477e00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64780aa930 0x7f6478152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f6477e00640 1 -- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64780cdc40 msgr2=0x7f6478152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:28.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f6477e00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64780cdc40 0x7f6478152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f6477e00640 1 -- 172.21.15.163:0/622414710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64780aaef0 con 0x7f64780b4420 2024-07-26T06:16:28.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f6477e00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 0x7f64780aa3f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f646800c450 tx=0x7f646800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64680052d0 con 0x7f64780b4420 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6468005470 con 0x7f64780b4420 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468012870 con 0x7f64780b4420 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.495+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64780ac900 con 0x7f64780b4420 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.496+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64780ac5b0 con 0x7f64780b4420 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.497+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64780cd530 con 0x7f64780b4420 2024-07-26T06:16:28.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.498+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f64680035f0 con 0x7f64780b4420 2024-07-26T06:16:28.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.499+0000 7f646fe00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6454076510 0x7f64540789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:28.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.499+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f6468097830 con 0x7f64780b4420 2024-07-26T06:16:28.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.499+0000 7f6477400640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6454076510 0x7f64540789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:28.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.501+0000 7f6477400640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6454076510 0x7f64540789d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f64780b07c0 tx=0x7f6460009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:28.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.503+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6468060de0 con 0x7f64780b4420 2024-07-26T06:16:28.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.749+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6478156f20 con 0x7f64780b4420 2024-07-26T06:16:28.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.750+0000 7f646fe00640 1 -- 172.21.15.163:0/622414710 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f64680606e0 con 0x7f64780b4420 2024-07-26T06:16:28.751 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:28.751 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:28.751 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:28.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6454076510 msgr2=0x7f64540789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:28.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6454076510 0x7f64540789d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f64780b07c0 tx=0x7f6460009040 comp rx=0 tx=0).stop 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 msgr2=0x7f64780aa3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 0x7f64780aa3f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f646800c450 tx=0x7f646800c910 comp rx=0 tx=0).stop 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 shutdown_connections 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6454076510 0x7f64540789d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64780aa930 0x7f6478152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64780cdc40 0x7f6478152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 --2- 172.21.15.163:0/622414710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64780b4420 0x7f64780aa3f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.753+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 >> 172.21.15.163:0/622414710 conn(0x7f64780b0e30 msgr2=0x7f64780aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.754+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 shutdown_connections 2024-07-26T06:16:28.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:28.754+0000 7f647dc00640 1 -- 172.21.15.163:0/622414710 wait complete. 2024-07-26T06:16:28.775 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:29.002 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":64.626918356999994,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.949435303913699,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:29.006 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.9494353039137 }, "state": "open", "uptime": 64.626918357 } ] 2024-07-26T06:16:29.006 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:34.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 -- 172.21.15.163:0/780924186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 msgr2=0x7ff0d8071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 --2- 172.21.15.163:0/780924186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d8071530 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff0d806ce80 tx=0x7ff0c002a530 comp rx=0 tx=0).stop 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 -- 172.21.15.163:0/780924186 shutdown_connections 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 --2- 172.21.15.163:0/780924186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d8071530 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 --2- 172.21.15.163:0/780924186 >> v2:172.21.15.164:3300/0 conn(0x7ff0d81556f0 0x7ff0d80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 -- 172.21.15.163:0/780924186 >> 172.21.15.163:0/780924186 conn(0x7ff0d80b0e30 msgr2=0x7ff0d80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 -- 172.21.15.163:0/780924186 shutdown_connections 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.225+0000 7ff0dca00640 1 -- 172.21.15.163:0/780924186 wait complete. 2024-07-26T06:16:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.227+0000 7ff0dca00640 1 Processor -- start 2024-07-26T06:16:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.227+0000 7ff0dca00640 1 -- start start 2024-07-26T06:16:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0dca00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0d8071a70 0x7ff0d814cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0dca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d81505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0dca00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0d81556f0 0x7ff0d813d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0dca00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8072000 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0dca00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8071e80 con 0x7ff0d81556f0 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0dca00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d815f310 con 0x7ff0d8071a70 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0cde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d81505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0cde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d81505a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52402/0 (socket says 172.21.15.163:52402) 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0cde00640 1 -- 172.21.15.163:0/1371795652 learned_addr learned my addr 172.21.15.163:0/1371795652 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0d6800640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0d8071a70 0x7ff0d814cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.228+0000 7ff0d7200640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0d81556f0 0x7ff0d813d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cde00640 1 -- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0d8071a70 msgr2=0x7ff0d814cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cde00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0d8071a70 0x7ff0d814cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cde00640 1 -- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0d81556f0 msgr2=0x7ff0d813d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cde00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0d81556f0 0x7ff0d813d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cde00640 1 -- 172.21.15.163:0/1371795652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d814ec10 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cde00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d81505a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009780 tx=0x7ff0c0032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0040070 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0c0005310 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d814eef0 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00047a0 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.229+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d814f790 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.230+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d814f2e0 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.232+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7ff0c0039040 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.232+0000 7ff0cf400640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0b4076510 0x7ff0b40789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.233+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7ff0c00b9050 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.233+0000 7ff0d6800640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0b4076510 0x7ff0b40789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:34.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.234+0000 7ff0d6800640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0b4076510 0x7ff0b40789d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005fd0 tx=0x7ff0c80097d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:34.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.237+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff0c00825d0 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.487+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0d815f850 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.489+0000 7ff0cf400640 1 -- 172.21.15.163:0/1371795652 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7ff0c0081ed0 con 0x7ff0d80ce2d0 2024-07-26T06:16:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:34.490 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:34.490 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0b4076510 msgr2=0x7ff0b40789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0b4076510 0x7ff0b40789d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005fd0 tx=0x7ff0c80097d0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 msgr2=0x7ff0d81505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d81505a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009780 tx=0x7ff0c0032c10 comp rx=0 tx=0).stop 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 shutdown_connections 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0b4076510 0x7ff0b40789d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0d81556f0 0x7ff0d813d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0d80ce2d0 0x7ff0d81505a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 --2- 172.21.15.163:0/1371795652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0d8071a70 0x7ff0d814cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 >> 172.21.15.163:0/1371795652 conn(0x7ff0d80b0e30 msgr2=0x7ff0d80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:34.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 shutdown_connections 2024-07-26T06:16:34.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:34.492+0000 7ff0dca00640 1 -- 172.21.15.163:0/1371795652 wait complete. 2024-07-26T06:16:34.513 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:34.746 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":70.368822187999996,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.6997006737354,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:34.746 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.6997006737354 }, "state": "open", "uptime": 70.368822188 } ] 2024-07-26T06:16:34.747 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:39.748 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.965+0000 7f130ba00640 1 -- 172.21.15.163:0/4278221773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12e8004800 msgr2=0x7f12e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.965+0000 7f130ba00640 1 --2- 172.21.15.163:0/4278221773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12e8004800 0x7f12e8004c80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00b9d0 tx=0x7f12e8005830 comp rx=0 tx=0).stop 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.965+0000 7f1309a00640 1 -- 172.21.15.163:0/4278221773 reap_dead start 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.966+0000 7f130ba00640 1 -- 172.21.15.163:0/4278221773 shutdown_connections 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.966+0000 7f130ba00640 1 -- 172.21.15.163:0/4278221773 >> 172.21.15.163:0/4278221773 conn(0x7f130406d420 msgr2=0x7f130406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.966+0000 7f130ba00640 1 -- 172.21.15.163:0/4278221773 shutdown_connections 2024-07-26T06:16:39.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.966+0000 7f130ba00640 1 -- 172.21.15.163:0/4278221773 wait complete. 2024-07-26T06:16:39.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.967+0000 7f130ba00640 1 Processor -- start 2024-07-26T06:16:39.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.967+0000 7f130ba00640 1 -- start start 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.968+0000 7f130ba00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12e8004800 0x7f13041504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.968+0000 7f130ba00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 0x7f13040adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.968+0000 7f130ba00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f130414cb90 0x7f130414cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.968+0000 7f130ba00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f130406c3b0 con 0x7f130414dd30 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.968+0000 7f130ba00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1304159680 con 0x7f12e8004800 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.968+0000 7f130ba00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f130406c530 con 0x7f130414cb90 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 0x7f13040adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 0x7f13040adb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43266/0 (socket says 172.21.15.163:43266) 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 -- 172.21.15.163:0/257238804 learned_addr learned my addr 172.21.15.163:0/257238804 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 -- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f130414cb90 msgr2=0x7f130414cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:39.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f130414cb90 0x7f130414cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:39.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 -- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12e8004800 msgr2=0x7f13041504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:39.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12e8004800 0x7f13041504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:39.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 -- 172.21.15.163:0/257238804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130414d530 con 0x7f130414dd30 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.969+0000 7f1303e00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 0x7f13040adb50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f12ec037040 tx=0x7f12ec034e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.970+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec043030 con 0x7f130414dd30 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.970+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12ec034470 con 0x7f130414dd30 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.970+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec039040 con 0x7f130414dd30 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.970+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1304150d70 con 0x7f130414dd30 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.970+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13040b09a0 con 0x7f130414dd30 2024-07-26T06:16:39.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.971+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13040b40a0 con 0x7f130414dd30 2024-07-26T06:16:39.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.972+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f12ec046070 con 0x7f130414dd30 2024-07-26T06:16:39.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.973+0000 7f1300c00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f12e0076510 0x7f12e00789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:39.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.973+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f12ec0b7b90 con 0x7f130414dd30 2024-07-26T06:16:39.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.974+0000 7f1309000640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f12e0076510 0x7f12e00789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:39.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.975+0000 7f1309000640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f12e0076510 0x7f12e00789d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f12f4006110 tx=0x7f12f4004a90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:39.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:39.978+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f12ec030050 con 0x7f130414dd30 2024-07-26T06:16:40.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.218+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13041509f0 con 0x7f130414dd30 2024-07-26T06:16:40.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.220+0000 7f1300c00640 1 -- 172.21.15.163:0/257238804 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f12ec034640 con 0x7f130414dd30 2024-07-26T06:16:40.221 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:40.221 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:40.221 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:40.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f12e0076510 msgr2=0x7f12e00789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:40.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f12e0076510 0x7f12e00789d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f12f4006110 tx=0x7f12f4004a90 comp rx=0 tx=0).stop 2024-07-26T06:16:40.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 msgr2=0x7f13040adb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 0x7f13040adb50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f12ec037040 tx=0x7f12ec034e80 comp rx=0 tx=0).stop 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 shutdown_connections 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f12e0076510 0x7f12e00789d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f130414cb90 0x7f130414cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f130414dd30 0x7f13040adb50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 --2- 172.21.15.163:0/257238804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12e8004800 0x7f13041504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.222+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 >> 172.21.15.163:0/257238804 conn(0x7f130406d420 msgr2=0x7f130415b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.223+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 shutdown_connections 2024-07-26T06:16:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:40.223+0000 7f130ba00640 1 -- 172.21.15.163:0/257238804 wait complete. 2024-07-26T06:16:40.243 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:40.473 INFO:teuthology.orchestra.run.smithi164.stdout:[{"id":4760,"entity":{"name":{"type":"client","num":4760},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":362720927}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":76.097726234999996,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.453811564664999,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.4760 172.21.15.164:0/362720927","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:16:40.474 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 362720927, "type": "any" }, "name": { "num": 4760, "type": "client" } }, "id": 4760, "inst": "client.4760 172.21.15.164:0/362720927", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.453811564665 }, "state": "open", "uptime": 76.097726235 } ] 2024-07-26T06:16:40.474 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-07-26T06:16:45.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f913d400640 1 -- 172.21.15.163:0/2842545756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f911c004800 msgr2=0x7f911c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f913d400640 1 --2- 172.21.15.163:0/2842545756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f911c004800 0x7f911c004c80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f912400baf0 tx=0x7f911c005830 comp rx=0 tx=0).stop 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f9137200640 1 -- 172.21.15.163:0/2842545756 reap_dead start 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f913d400640 1 -- 172.21.15.163:0/2842545756 shutdown_connections 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f913d400640 1 -- 172.21.15.163:0/2842545756 >> 172.21.15.163:0/2842545756 conn(0x7f913806d420 msgr2=0x7f913806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f913d400640 1 -- 172.21.15.163:0/2842545756 shutdown_connections 2024-07-26T06:16:45.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.695+0000 7f913d400640 1 -- 172.21.15.163:0/2842545756 wait complete. 2024-07-26T06:16:45.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.697+0000 7f913d400640 1 Processor -- start 2024-07-26T06:16:45.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.697+0000 7f913d400640 1 -- start start 2024-07-26T06:16:45.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f913d400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f911c004800 0x7f9138143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:45.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f913d400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 0x7f91380ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:45.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f913d400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91380ade20 0x7f91381e2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:45.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f913d400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f913815bbc0 con 0x7f9138146de0 2024-07-26T06:16:45.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f913d400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f913815ba40 con 0x7f91380ade20 2024-07-26T06:16:45.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f913d400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f913806c3d0 con 0x7f911c004800 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f9135e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 0x7f91380ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f9135e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 0x7f91380ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34942/0 (socket says 172.21.15.163:34942) 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.698+0000 7f9135e00640 1 -- 172.21.15.163:0/2658648742 learned_addr learned my addr 172.21.15.163:0/2658648742 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9136800640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f911c004800 0x7f9138143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9137200640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91380ade20 0x7f91381e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9135e00640 1 -- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f911c004800 msgr2=0x7f9138143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9135e00640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f911c004800 0x7f9138143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:45.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9135e00640 1 -- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91380ade20 msgr2=0x7f91381e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:45.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9135e00640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91380ade20 0x7f91381e2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:45.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9135e00640 1 -- 172.21.15.163:0/2658648742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9138156250 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f9135e00640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 0x7f91380ad8e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f912400bbe0 tx=0x7f9124032720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124004230 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91240044c0 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124032b70 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.699+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9138156530 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.700+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9138156af0 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.701+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91380b34e0 con 0x7f9138146de0 2024-07-26T06:16:45.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.702+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 8) ==== 98405+0+0 (secure 0 0 0) 0x7f912404c060 con 0x7f9138146de0 2024-07-26T06:16:45.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.703+0000 7f912ea00640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9118076510 0x7f91180789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:45.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.703+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f91240b8780 con 0x7f9138146de0 2024-07-26T06:16:45.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.704+0000 7f9136800640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9118076510 0x7f91180789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:45.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.705+0000 7f9136800640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9118076510 0x7f91180789d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9128000c00 tx=0x7f9128006b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.707+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9124040070 con 0x7f9138146de0 2024-07-26T06:16:45.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.952+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f913815d9d0 con 0x7f9138146de0 2024-07-26T06:16:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.954+0000 7f912ea00640 1 -- 172.21.15.163:0/2658648742 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f9124081b00 con 0x7f9138146de0 2024-07-26T06:16:45.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:45.955 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9118076510 msgr2=0x7f91180789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9118076510 0x7f91180789d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9128000c00 tx=0x7f9128006b50 comp rx=0 tx=0).stop 2024-07-26T06:16:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 msgr2=0x7f91380ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 0x7f91380ad8e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f912400bbe0 tx=0x7f9124032720 comp rx=0 tx=0).stop 2024-07-26T06:16:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 shutdown_connections 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9118076510 0x7f91180789d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91380ade20 0x7f91381e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9138146de0 0x7f91380ad8e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 --2- 172.21.15.163:0/2658648742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f911c004800 0x7f9138143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.956+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 >> 172.21.15.163:0/2658648742 conn(0x7f913806d420 msgr2=0x7f91380cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.957+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 shutdown_connections 2024-07-26T06:16:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:45.957+0000 7f913d400640 1 -- 172.21.15.163:0/2658648742 wait complete. 2024-07-26T06:16:45.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-07-26T06:16:46.216 INFO:teuthology.orchestra.run.smithi164.stdout:[] 2024-07-26T06:16:46.217 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-07-26T06:16:46.217 DEBUG:tasks.ceph_test_case:wait_until_true: success in 70s and 0 retries 2024-07-26T06:16:46.218 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:16:46.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:16:46.354 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:16:46.471 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:16:46.471 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:16:46.471 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:16:46.472 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:46.528 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:16:46.528 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:16:46.528 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:16:46.528 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:46.585 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:16:46.585 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:16:46.585 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:16:46.585 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:46.642 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:16:46.643 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:16:46.643 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:16:46.643 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:46.701 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:16:46.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 -- 172.21.15.163:0/300985122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 msgr2=0x7f9e7c0ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 --2- 172.21.15.163:0/300985122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c0ce140 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0aec80 tx=0x7f9e7402a9c0 comp rx=0 tx=0).stop 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 -- 172.21.15.163:0/300985122 shutdown_connections 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 --2- 172.21.15.163:0/300985122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c0ce140 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 --2- 172.21.15.163:0/300985122 >> v2:172.21.15.164:3300/0 conn(0x7f9e7c1404a0 0x7f9e7c161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 -- 172.21.15.163:0/300985122 >> 172.21.15.163:0/300985122 conn(0x7f9e7c06d420 msgr2=0x7f9e7c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 -- 172.21.15.163:0/300985122 shutdown_connections 2024-07-26T06:16:46.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.914+0000 7f9e81200640 1 -- 172.21.15.163:0/300985122 wait complete. 2024-07-26T06:16:46.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.915+0000 7f9e81200640 1 Processor -- start 2024-07-26T06:16:46.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.916+0000 7f9e81200640 1 -- start start 2024-07-26T06:16:46.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e81200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e7c06bf40 0x7f9e7c137e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e81200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c1398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e81200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e7c1404a0 0x7f9e7c139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e81200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c0ce680 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e81200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c161ec0 con 0x7f9e7c1404a0 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e81200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9e7c0ce800 con 0x7f9e7c06bf40 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c1398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c1398b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34968/0 (socket says 172.21.15.163:34968) 2024-07-26T06:16:46.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 -- 172.21.15.163:0/4047083780 learned_addr learned my addr 172.21.15.163:0/4047083780 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:46.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79e00640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e7c06bf40 0x7f9e7c137e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:46.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 -- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e7c06bf40 msgr2=0x7f9e7c137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e7c06bf40 0x7f9e7c137e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e7a800640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e7c1404a0 0x7f9e7c139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 -- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e7c1404a0 msgr2=0x7f9e7c139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e7c1404a0 0x7f9e7c139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.917+0000 7f9e79400640 1 -- 172.21.15.163:0/4047083780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7c138380 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.918+0000 7f9e79400640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c1398b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c760 tx=0x7f9e6800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.918+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68017070 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.918+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e68005380 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.918+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7c0b1e40 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.918+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800a8a0 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.918+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7c0715c0 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.919+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e7c0cecf0 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.920+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f9e6801a070 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.922+0000 7f9e72000640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9e5c0764b0 0x7f9e5c078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:46.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.922+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f9e680970f0 con 0x7f9e7c0cdd10 2024-07-26T06:16:46.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.922+0000 7f9e79e00640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9e5c0764b0 0x7f9e5c078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:46.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.924+0000 7f9e79e00640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9e5c0764b0 0x7f9e5c078970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0058c0 tx=0x7f9e6c005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:46.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:46.925+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9e680606f0 con 0x7f9e7c0cdd10 2024-07-26T06:16:47.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.167+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e7c06cd80 con 0x7f9e7c0cdd10 2024-07-26T06:16:47.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.168+0000 7f9e72000640 1 -- 172.21.15.163:0/4047083780 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f9e6805fff0 con 0x7f9e7c0cdd10 2024-07-26T06:16:47.169 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:47.169 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:47.169 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:47.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.170+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9e5c0764b0 msgr2=0x7f9e5c078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.170+0000 7f9e81200640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9e5c0764b0 0x7f9e5c078970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0058c0 tx=0x7f9e6c005850 comp rx=0 tx=0).stop 2024-07-26T06:16:47.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.170+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 msgr2=0x7f9e7c1398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.170+0000 7f9e81200640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c1398b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c760 tx=0x7f9e6800cc20 comp rx=0 tx=0).stop 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 shutdown_connections 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9e5c0764b0 0x7f9e5c078970 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e7c1404a0 0x7f9e7c139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e7c0cdd10 0x7f9e7c1398b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 --2- 172.21.15.163:0/4047083780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e7c06bf40 0x7f9e7c137e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 >> 172.21.15.163:0/4047083780 conn(0x7f9e7c06d420 msgr2=0x7f9e7c140b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 shutdown_connections 2024-07-26T06:16:47.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.171+0000 7f9e81200640 1 -- 172.21.15.163:0/4047083780 wait complete. 2024-07-26T06:16:47.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 -- 172.21.15.163:0/4023821126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cbad0 msgr2=0x7f11700cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 --2- 172.21.15.163:0/4023821126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cbad0 0x7f11700cd530 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f11700b1bc0 tx=0x7f116002a560 comp rx=0 tx=0).stop 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 -- 172.21.15.163:0/4023821126 shutdown_connections 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 --2- 172.21.15.163:0/4023821126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11700cd000 0x7f117015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 --2- 172.21.15.163:0/4023821126 >> v2:172.21.15.164:3300/0 conn(0x7f11700b4420 0x7f11700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 --2- 172.21.15.163:0/4023821126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cbad0 0x7f11700cd530 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 -- 172.21.15.163:0/4023821126 >> 172.21.15.163:0/4023821126 conn(0x7f11700b0e30 msgr2=0x7f11700b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:47.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 -- 172.21.15.163:0/4023821126 shutdown_connections 2024-07-26T06:16:47.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.389+0000 7f116fe00640 1 -- 172.21.15.163:0/4023821126 wait complete. 2024-07-26T06:16:47.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.391+0000 7f116fe00640 1 Processor -- start 2024-07-26T06:16:47.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.391+0000 7f116fe00640 1 -- start start 2024-07-26T06:16:47.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.392+0000 7f116fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11700b4420 0x7f1170157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.392+0000 7f116fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 0x7f11701436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.392+0000 7f116fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1170156240 0x7f11700ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.392+0000 7f116fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f117015d510 con 0x7f11700cd000 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.392+0000 7f116fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f117015d390 con 0x7f11700b4420 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.392+0000 7f116fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f117015d690 con 0x7f1170156240 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 0x7f11701436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 0x7f11701436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34988/0 (socket says 172.21.15.163:34988) 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 -- 172.21.15.163:0/2265064500 learned_addr learned my addr 172.21.15.163:0/2265064500 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f116ea00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11700b4420 0x7f1170157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f116f400640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1170156240 0x7f11700ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 -- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1170156240 msgr2=0x7f11700ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1170156240 0x7f11700ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 -- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11700b4420 msgr2=0x7f1170157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11700b4420 0x7f1170157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 -- 172.21.15.163:0/2265064500 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11701580b0 con 0x7f11700cd000 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f116f400640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1170156240 0x7f11700ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.393+0000 7f1167e00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 0x7f11701436f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f115800b4a0 tx=0x7f115800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.394+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158014070 con 0x7f11700cd000 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.394+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1158010700 con 0x7f11700cd000 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.394+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158010b50 con 0x7f11700cd000 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.394+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11701f7340 con 0x7f11700cd000 2024-07-26T06:16:47.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.394+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11701f7900 con 0x7f11700cd000 2024-07-26T06:16:47.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.395+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f117015ce50 con 0x7f11700cd000 2024-07-26T06:16:47.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.396+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f1158017070 con 0x7f11700cd000 2024-07-26T06:16:47.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.397+0000 7f1166a00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f114c076480 0x7f114c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.397+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f1158097650 con 0x7f11700cd000 2024-07-26T06:16:47.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.398+0000 7f116ea00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f114c076480 0x7f114c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.399+0000 7f116ea00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f114c076480 0x7f114c078940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1160033000 tx=0x7f1160004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:47.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.401+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1158060c60 con 0x7f11700cd000 2024-07-26T06:16:47.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.632+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11700b13b0 con 0x7f11700cd000 2024-07-26T06:16:47.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.633+0000 7f1166a00640 1 -- 172.21.15.163:0/2265064500 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f1158060560 con 0x7f11700cd000 2024-07-26T06:16:47.634 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:47.634 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:47.635 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:47.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f114c076480 msgr2=0x7f114c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f114c076480 0x7f114c078940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1160033000 tx=0x7f1160004730 comp rx=0 tx=0).stop 2024-07-26T06:16:47.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 msgr2=0x7f11701436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 0x7f11701436f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f115800b4a0 tx=0x7f115800b960 comp rx=0 tx=0).stop 2024-07-26T06:16:47.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 shutdown_connections 2024-07-26T06:16:47.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f114c076480 0x7f114c078940 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1170156240 0x7f11700ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11700cd000 0x7f11701436f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 --2- 172.21.15.163:0/2265064500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11700b4420 0x7f1170157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 >> 172.21.15.163:0/2265064500 conn(0x7f11700b0e30 msgr2=0x7f1170073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:47.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 shutdown_connections 2024-07-26T06:16:47.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.636+0000 7f116fe00640 1 -- 172.21.15.163:0/2265064500 wait complete. 2024-07-26T06:16:47.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:16:47.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.851+0000 7f8e77000640 1 -- 172.21.15.163:0/2654345716 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e701556f0 msgr2=0x7f8e700cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.851+0000 7f8e77000640 1 --2- 172.21.15.163:0/2654345716 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e701556f0 0x7f8e700cdd90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8e700ced10 tx=0x7f8e6402a560 comp rx=0 tx=0).stop 2024-07-26T06:16:47.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.852+0000 7f8e77000640 1 -- 172.21.15.163:0/2654345716 shutdown_connections 2024-07-26T06:16:47.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.852+0000 7f8e77000640 1 --2- 172.21.15.163:0/2654345716 >> v2:172.21.15.164:3300/0 conn(0x7f8e70071c40 0x7f8e700b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.852+0000 7f8e77000640 1 --2- 172.21.15.163:0/2654345716 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e701556f0 0x7f8e700cdd90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.852+0000 7f8e77000640 1 -- 172.21.15.163:0/2654345716 >> 172.21.15.163:0/2654345716 conn(0x7f8e700b0e30 msgr2=0x7f8e700b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.852+0000 7f8e77000640 1 -- 172.21.15.163:0/2654345716 shutdown_connections 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.852+0000 7f8e77000640 1 -- 172.21.15.163:0/2654345716 wait complete. 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.853+0000 7f8e77000640 1 Processor -- start 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.853+0000 7f8e77000640 1 -- start start 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.854+0000 7f8e77000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e70071c40 0x7f8e7015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.854+0000 7f8e77000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 0x7f8e700cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.854+0000 7f8e77000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e701556f0 0x7f8e700cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.854+0000 7f8e77000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e700b4020 con 0x7f8e700ce2d0 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.854+0000 7f8e77000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8e700b3ea0 con 0x7f8e70071c40 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.854+0000 7f8e77000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8e700b41a0 con 0x7f8e701556f0 2024-07-26T06:16:47.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 0x7f8e700cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 0x7f8e700cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35010/0 (socket says 172.21.15.163:35010) 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 -- 172.21.15.163:0/1399825596 learned_addr learned my addr 172.21.15.163:0/1399825596 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75c00640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e70071c40 0x7f8e7015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e76600640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e701556f0 0x7f8e700cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 -- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e701556f0 msgr2=0x7f8e700cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e701556f0 0x7f8e700cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 -- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e70071c40 msgr2=0x7f8e7015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e70071c40 0x7f8e7015ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.855+0000 7f8e75200640 1 -- 172.21.15.163:0/1399825596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e700af2c0 con 0x7f8e700ce2d0 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.856+0000 7f8e75200640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 0x7f8e700cbb80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00c430 tx=0x7f8e5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.856+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c011790 con 0x7f8e700ce2d0 2024-07-26T06:16:47.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.856+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e5c017070 con 0x7f8e700ce2d0 2024-07-26T06:16:47.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.856+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c011d00 con 0x7f8e700ce2d0 2024-07-26T06:16:47.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.856+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7015a070 con 0x7f8e700ce2d0 2024-07-26T06:16:47.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.856+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7015a310 con 0x7f8e700ce2d0 2024-07-26T06:16:47.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.857+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e700b4730 con 0x7f8e700ce2d0 2024-07-26T06:16:47.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.858+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f8e5c01c460 con 0x7f8e700ce2d0 2024-07-26T06:16:47.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.859+0000 7f8e6d600640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8e50076480 0x7f8e50078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:47.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.859+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f8e5c097a90 con 0x7f8e700ce2d0 2024-07-26T06:16:47.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.860+0000 7f8e75c00640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8e50076480 0x7f8e50078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:47.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.861+0000 7f8e75c00640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8e50076480 0x7f8e50078940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8e64033000 tx=0x7f8e64004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:47.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:47.863+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8e5c061120 con 0x7f8e700ce2d0 2024-07-26T06:16:48.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.073+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e700b34f0 con 0x7f8e700ce2d0 2024-07-26T06:16:48.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.075+0000 7f8e6d600640 1 -- 172.21.15.163:0/1399825596 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+16729 (secure 0 0 0) 0x7f8e5c069110 con 0x7f8e700ce2d0 2024-07-26T06:16:48.077 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:48.077 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":47,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:15:15.448534+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:15:04.734772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-07-26T06:15:05.851248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":5}]},{"pgid":"7.35","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8e50076480 msgr2=0x7f8e50078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8e50076480 0x7f8e50078940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8e64033000 tx=0x7f8e64004730 comp rx=0 tx=0).stop 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 msgr2=0x7f8e700cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 0x7f8e700cbb80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00c430 tx=0x7f8e5c00c8f0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 shutdown_connections 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8e50076480 0x7f8e50078940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e701556f0 0x7f8e700cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e700ce2d0 0x7f8e700cbb80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 --2- 172.21.15.163:0/1399825596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e70071c40 0x7f8e7015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 >> 172.21.15.163:0/1399825596 conn(0x7f8e700b0e30 msgr2=0x7f8e7014f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:48.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 shutdown_connections 2024-07-26T06:16:48.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.078+0000 7f8e77000640 1 -- 172.21.15.163:0/1399825596 wait complete. 2024-07-26T06:16:48.097 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:16:48.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 -- 172.21.15.163:0/1153729786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42d0003a50 msgr2=0x7f42d0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 --2- 172.21.15.163:0/1153729786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42d0003a50 0x7f42d0003e70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00c2e0 tx=0x7f42d0005830 comp rx=0 tx=0).stop 2024-07-26T06:16:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 -- 172.21.15.163:0/1153729786 shutdown_connections 2024-07-26T06:16:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 --2- 172.21.15.163:0/1153729786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42d0003a50 0x7f42d0003e70 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 -- 172.21.15.163:0/1153729786 >> 172.21.15.163:0/1153729786 conn(0x7f42ec0cfda0 msgr2=0x7f42ec0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 -- 172.21.15.163:0/1153729786 shutdown_connections 2024-07-26T06:16:48.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.297+0000 7f42f2e00640 1 -- 172.21.15.163:0/1153729786 wait complete. 2024-07-26T06:16:48.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.299+0000 7f42f2e00640 1 Processor -- start 2024-07-26T06:16:48.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.299+0000 7f42f2e00640 1 -- start start 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42d0003a50 0x7f42ec1f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 0x7f42ec1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42ec0cc5c0 0x7f42ec0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec0ce680 con 0x7f42ec0c97a0 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec0ce500 con 0x7f42d0003a50 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42ec0ce800 con 0x7f42ec0cc5c0 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 0x7f42ec1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 0x7f42ec1585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35026/0 (socket says 172.21.15.163:35026) 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 -- 172.21.15.163:0/3980720760 learned_addr learned my addr 172.21.15.163:0/3980720760 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:48.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 -- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42ec0cc5c0 msgr2=0x7f42ec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f2400640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42ec0cc5c0 0x7f42ec0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42ec0cc5c0 0x7f42ec0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 -- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42d0003a50 msgr2=0x7f42ec1f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42d0003a50 0x7f42ec1f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.300+0000 7f42f1000640 1 -- 172.21.15.163:0/3980720760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec1f7f80 con 0x7f42ec0c97a0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.301+0000 7f42f1000640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 0x7f42ec1585e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f42d800d630 tx=0x7f42d800daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.301+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8017070 con 0x7f42ec0c97a0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.301+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42d8005310 con 0x7f42ec0c97a0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.301+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42ec1f8260 con 0x7f42ec0c97a0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.301+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d80047a0 con 0x7f42ec0c97a0 2024-07-26T06:16:48.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.301+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42ec15db00 con 0x7f42ec0c97a0 2024-07-26T06:16:48.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.302+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42ec0b6740 con 0x7f42ec0c97a0 2024-07-26T06:16:48.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.303+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f42d8004940 con 0x7f42ec0c97a0 2024-07-26T06:16:48.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.305+0000 7f42e9600640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f42cc0764b0 0x7f42cc078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.305+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f42d8097a30 con 0x7f42ec0c97a0 2024-07-26T06:16:48.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.305+0000 7f42f1a00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f42cc0764b0 0x7f42cc078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:48.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.306+0000 7f42f1a00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f42cc0764b0 0x7f42cc078970 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f42dc033000 tx=0x7f42dc001a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:48.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.309+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f42d8060fb0 con 0x7f42ec0c97a0 2024-07-26T06:16:48.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.548+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42ec159eb0 con 0x7f42ec0c97a0 2024-07-26T06:16:48.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.549+0000 7f42e9600640 1 -- 172.21.15.163:0/3980720760 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7f42d80608b0 con 0x7f42ec0c97a0 2024-07-26T06:16:48.551 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:48.551 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:48.551 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:48.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f42cc0764b0 msgr2=0x7f42cc078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f42cc0764b0 0x7f42cc078970 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f42dc033000 tx=0x7f42dc001a30 comp rx=0 tx=0).stop 2024-07-26T06:16:48.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 msgr2=0x7f42ec1585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 0x7f42ec1585e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f42d800d630 tx=0x7f42d800daf0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 shutdown_connections 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f42cc0764b0 0x7f42cc078970 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42ec0cc5c0 0x7f42ec0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42ec0c97a0 0x7f42ec1585e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 --2- 172.21.15.163:0/3980720760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42d0003a50 0x7f42ec1f7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 >> 172.21.15.163:0/3980720760 conn(0x7f42ec0cfda0 msgr2=0x7f42ec0b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.552+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 shutdown_connections 2024-07-26T06:16:48.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.553+0000 7f42f2e00640 1 -- 172.21.15.163:0/3980720760 wait complete. 2024-07-26T06:16:48.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.772+0000 7ff1d4a00640 1 -- 172.21.15.163:0/419989359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d015d3a0 msgr2=0x7ff1d00b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.772+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/419989359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d015d3a0 0x7ff1d00b5120 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b3c90 tx=0x7ff1c002a530 comp rx=0 tx=0).stop 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.772+0000 7ff1d4a00640 1 -- 172.21.15.163:0/419989359 shutdown_connections 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.772+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/419989359 >> v2:172.21.15.164:3300/0 conn(0x7ff1d0071130 0x7ff1d00ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.772+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/419989359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d015d3a0 0x7ff1d00b5120 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.772+0000 7ff1d4a00640 1 -- 172.21.15.163:0/419989359 >> 172.21.15.163:0/419989359 conn(0x7ff1d00b3060 msgr2=0x7ff1d00b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.773+0000 7ff1d4a00640 1 -- 172.21.15.163:0/419989359 shutdown_connections 2024-07-26T06:16:48.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.773+0000 7ff1d4a00640 1 -- 172.21.15.163:0/419989359 wait complete. 2024-07-26T06:16:48.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.774+0000 7ff1d4a00640 1 Processor -- start 2024-07-26T06:16:48.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.774+0000 7ff1d4a00640 1 -- start start 2024-07-26T06:16:48.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.775+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 0x7ff1d015f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.775+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d00b5660 0x7ff1d0154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.775+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d015d3a0 0x7ff1d00ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.775+0000 7ff1d4a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d00cf0b0 con 0x7ff1d0071130 2024-07-26T06:16:48.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.775+0000 7ff1d4a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d00cef30 con 0x7ff1d015d3a0 2024-07-26T06:16:48.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.775+0000 7ff1d4a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d00cf230 con 0x7ff1d00b5660 2024-07-26T06:16:48.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 0x7ff1d015f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 0x7ff1d015f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35040/0 (socket says 172.21.15.163:35040) 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 -- 172.21.15.163:0/745619139 learned_addr learned my addr 172.21.15.163:0/745619139 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 -- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d00b5660 msgr2=0x7ff1d0154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d00b5660 0x7ff1d0154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 -- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d015d3a0 msgr2=0x7ff1d00ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d015d3a0 0x7ff1d00ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 -- 172.21.15.163:0/745619139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d0160560 con 0x7ff1d0071130 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.776+0000 7ff1ce800640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 0x7ff1d015f680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff1c002ab20 tx=0x7ff1c0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.777+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0005c90 con 0x7ff1d0071130 2024-07-26T06:16:48.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.777+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1c0031e50 con 0x7ff1d0071130 2024-07-26T06:16:48.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.777+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d0160840 con 0x7ff1d0071130 2024-07-26T06:16:48.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.777+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0005240 con 0x7ff1d0071130 2024-07-26T06:16:48.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.777+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d015f160 con 0x7ff1d0071130 2024-07-26T06:16:48.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.778+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d015f360 con 0x7ff1d0071130 2024-07-26T06:16:48.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.779+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7ff1c00053e0 con 0x7ff1d0071130 2024-07-26T06:16:48.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.780+0000 7ff1c7400640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff1ac076480 0x7ff1ac078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:48.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.780+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7ff1c00b8b40 con 0x7ff1d0071130 2024-07-26T06:16:48.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.781+0000 7ff1c5e00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff1ac076480 0x7ff1ac078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:48.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.782+0000 7ff1c5e00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff1ac076480 0x7ff1ac078940 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80046b0 tx=0x7ff1b80012b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:48.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:48.786+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff1c003f030 con 0x7ff1d0071130 2024-07-26T06:16:49.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.028+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1d0160d20 con 0x7ff1d0071130 2024-07-26T06:16:49.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.029+0000 7ff1c7400640 1 -- 172.21.15.163:0/745619139 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5229 (secure 0 0 0) 0x7ff1c00820c0 con 0x7ff1d0071130 2024-07-26T06:16:49.030 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:49.030 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":26,"btime":"2024-07-26T06:15:20:736934+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:15:06.868107+0000","modified":"2024-07-26T06:15:11.906262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5256":{"gid":5256,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/1618066632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1618066632},{"type":"v1","addr":"172.21.15.164:6835","nonce":1618066632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5256,"qdb_cluster":[5256]},"id":3}]} 2024-07-26T06:16:49.031 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 26 2024-07-26T06:16:49.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.031+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff1ac076480 msgr2=0x7ff1ac078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.031+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff1ac076480 0x7ff1ac078940 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80046b0 tx=0x7ff1b80012b0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.031+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 msgr2=0x7ff1d015f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.031+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 0x7ff1d015f680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff1c002ab20 tx=0x7ff1c0033c20 comp rx=0 tx=0).stop 2024-07-26T06:16:49.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.031+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 shutdown_connections 2024-07-26T06:16:49.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff1ac076480 0x7ff1ac078940 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d015d3a0 0x7ff1d00ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d00b5660 0x7ff1d0154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 --2- 172.21.15.163:0/745619139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff1d0071130 0x7ff1d015f680 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 >> 172.21.15.163:0/745619139 conn(0x7ff1d00b3060 msgr2=0x7ff1d0070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:49.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 shutdown_connections 2024-07-26T06:16:49.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.032+0000 7ff1d4a00640 1 -- 172.21.15.163:0/745619139 wait complete. 2024-07-26T06:16:49.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:16:49.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 -- 172.21.15.163:0/3001843789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 msgr2=0x7f561c071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 --2- 172.21.15.163:0/3001843789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c071530 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f561c06cf00 tx=0x7f560402a9e0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 -- 172.21.15.163:0/3001843789 shutdown_connections 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 --2- 172.21.15.163:0/3001843789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c071a70 0x7f561c0b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 --2- 172.21.15.163:0/3001843789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c071530 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 -- 172.21.15.163:0/3001843789 >> 172.21.15.163:0/3001843789 conn(0x7f561c0b0e30 msgr2=0x7f561c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 -- 172.21.15.163:0/3001843789 shutdown_connections 2024-07-26T06:16:49.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.238+0000 7f5620a00640 1 -- 172.21.15.163:0/3001843789 wait complete. 2024-07-26T06:16:49.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.240+0000 7f5620a00640 1 Processor -- start 2024-07-26T06:16:49.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.240+0000 7f5620a00640 1 -- start start 2024-07-26T06:16:49.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.241+0000 7f5620a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c071a70 0x7f561c1f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.241+0000 7f5620a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.241+0000 7f5620a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c1556f0 0x7f561c1429b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.241+0000 7f5620a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0b3fe0 con 0x7f561c0ce4a0 2024-07-26T06:16:49.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.241+0000 7f5620a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0b3e60 con 0x7f561c1556f0 2024-07-26T06:16:49.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.241+0000 7f5620a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f561c0b4160 con 0x7f561c071a70 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c142470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35052/0 (socket says 172.21.15.163:35052) 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 -- 172.21.15.163:0/3887914184 learned_addr learned my addr 172.21.15.163:0/3887914184 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561b400640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c071a70 0x7f561c1f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 -- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c071a70 msgr2=0x7f561c1f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c071a70 0x7f561c1f42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 -- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c1556f0 msgr2=0x7f561c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c1556f0 0x7f561c1429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 -- 172.21.15.163:0/3887914184 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c142ef0 con 0x7f561c0ce4a0 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.242+0000 7f561aa00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c142470 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5604009bf0 tx=0x7f5604003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:49.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.243+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604047070 con 0x7f561c0ce4a0 2024-07-26T06:16:49.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.243+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5604004250 con 0x7f561c0ce4a0 2024-07-26T06:16:49.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.243+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c143160 con 0x7f561c0ce4a0 2024-07-26T06:16:49.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.243+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604033050 con 0x7f561c0ce4a0 2024-07-26T06:16:49.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.243+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c143720 con 0x7f561c0ce4a0 2024-07-26T06:16:49.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.244+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c0b4710 con 0x7f561c0ce4a0 2024-07-26T06:16:49.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.245+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f560404a070 con 0x7f561c0ce4a0 2024-07-26T06:16:49.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.247+0000 7f5613400640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f55f8076480 0x7f55f8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.247+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7f56040b8f70 con 0x7f561c0ce4a0 2024-07-26T06:16:49.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.247+0000 7f561b400640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f55f8076480 0x7f55f8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.248+0000 7f561b400640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f55f8076480 0x7f55f8078940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f560c00a730 tx=0x7f560c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:49.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.251+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f56040823e0 con 0x7f561c0ce4a0 2024-07-26T06:16:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.454+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f561c0b34f0 con 0x7f561c0ce4a0 2024-07-26T06:16:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.456+0000 7f5613400640 1 -- 172.21.15.163:0/3887914184 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+16729 (secure 0 0 0) 0x7f560408a3d0 con 0x7f561c0ce4a0 2024-07-26T06:16:49.457 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:49.458 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":47,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:15:15.448534+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:15:04.734772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-07-26T06:15:05.851248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.13","mappings":[{"from":0,"to":5}]},{"pgid":"7.35","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:16:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f55f8076480 msgr2=0x7f55f8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f55f8076480 0x7f55f8078940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f560c00a730 tx=0x7f560c008040 comp rx=0 tx=0).stop 2024-07-26T06:16:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 msgr2=0x7f561c142470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c142470 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5604009bf0 tx=0x7f5604003f10 comp rx=0 tx=0).stop 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 shutdown_connections 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f55f8076480 0x7f55f8078940 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c1556f0 0x7f561c1429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561c0ce4a0 0x7f561c142470 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 --2- 172.21.15.163:0/3887914184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c071a70 0x7f561c1f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 >> 172.21.15.163:0/3887914184 conn(0x7f561c0b0e30 msgr2=0x7f561c15d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.459+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 shutdown_connections 2024-07-26T06:16:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.460+0000 7f5620a00640 1 -- 172.21.15.163:0/3887914184 wait complete. 2024-07-26T06:16:49.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:16:49.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 -- 172.21.15.163:0/4213182197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec0ce830 msgr2=0x7fa8ec0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/4213182197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec0ce830 0x7fa8ec0b64c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0690b0 tx=0x7fa8d402aa30 comp rx=0 tx=0).stop 2024-07-26T06:16:49.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 -- 172.21.15.163:0/4213182197 shutdown_connections 2024-07-26T06:16:49.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/4213182197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec0ce830 0x7fa8ec0b64c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/4213182197 >> v2:172.21.15.164:3300/0 conn(0x7fa8ec15d5a0 0x7fa8ec0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 -- 172.21.15.163:0/4213182197 >> 172.21.15.163:0/4213182197 conn(0x7fa8ec0cfda0 msgr2=0x7fa8ec0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:49.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 -- 172.21.15.163:0/4213182197 shutdown_connections 2024-07-26T06:16:49.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.677+0000 7fa8ebe00640 1 -- 172.21.15.163:0/4213182197 wait complete. 2024-07-26T06:16:49.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.679+0000 7fa8ebe00640 1 Processor -- start 2024-07-26T06:16:49.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.679+0000 7fa8ebe00640 1 -- start start 2024-07-26T06:16:49.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ebe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8ec0b1e80 0x7fa8ec0b7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ebe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8ec0ce830 0x7fa8ec0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ebe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 0x7fa8ec0b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ebe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ec0b5130 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ebe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ec0b4fb0 con 0x7fa8ec0ce830 2024-07-26T06:16:49.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ebe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa8ec0b52b0 con 0x7fa8ec0b1e80 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8eb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 0x7fa8ec0b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8eb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 0x7fa8ec0b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35066/0 (socket says 172.21.15.163:35066) 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8eb400640 1 -- 172.21.15.163:0/3023961216 learned_addr learned my addr 172.21.15.163:0/3023961216 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8ea000640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8ec0ce830 0x7fa8ec0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.680+0000 7fa8eaa00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8ec0b1e80 0x7fa8ec0b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eb400640 1 -- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8ec0b1e80 msgr2=0x7fa8ec0b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eb400640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8ec0b1e80 0x7fa8ec0b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eb400640 1 -- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8ec0ce830 msgr2=0x7fa8ec0b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eb400640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8ec0ce830 0x7fa8ec0b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eb400640 1 -- 172.21.15.163:0/3023961216 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8ec1f3c80 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eaa00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8ec0b1e80 0x7fa8ec0b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8eb400640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 0x7fa8ec0b1760 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c400 tx=0x7fa8e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4017070 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8e40035f0 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8ec1f3f40 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4003a40 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.681+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8ec1f4500 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.682+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8ec050910 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.684+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fa8e401a070 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.685+0000 7fa8e2a00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa8c8076480 0x7fa8c8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.685+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5767+0+0 (secure 0 0 0) 0x7fa8e4096e90 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.686+0000 7fa8eaa00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa8c8076480 0x7fa8c8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.687+0000 7fa8eaa00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa8c8076480 0x7fa8c8078940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0058c0 tx=0x7fa8dc005cb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:49.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.690+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa8ec050910 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.935+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa8ec0b5820 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.937 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:16:49.936+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:16:49.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.940+0000 7fa8e2a00640 1 -- 172.21.15.163:0/3023961216 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7fa8e4098020 con 0x7fa8ec15d5a0 2024-07-26T06:16:49.941 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:16:49.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa8c8076480 msgr2=0x7fa8c8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa8c8076480 0x7fa8c8078940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0058c0 tx=0x7fa8dc005cb0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 msgr2=0x7fa8ec0b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 0x7fa8ec0b1760 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c400 tx=0x7fa8e400c8c0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 shutdown_connections 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa8c8076480 0x7fa8c8078940 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8ec15d5a0 0x7fa8ec0b1760 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8ec0ce830 0x7fa8ec0b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 --2- 172.21.15.163:0/3023961216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8ec0b1e80 0x7fa8ec0b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.943+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 >> 172.21.15.163:0/3023961216 conn(0x7fa8ec0cfda0 msgr2=0x7fa8ec0b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.944+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 shutdown_connections 2024-07-26T06:16:49.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:49.944+0000 7fa8ebe00640 1 -- 172.21.15.163:0/3023961216 wait complete. 2024-07-26T06:16:49.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:16:50.072 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.171+0000 7f383a400640 1 -- 172.21.15.163:0/3644296568 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3818003fd0 msgr2=0x7f3818004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.171+0000 7f383a400640 1 --2- 172.21.15.163:0/3644296568 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3818003fd0 0x7f3818004800 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f381c009ed0 tx=0x7f3818005830 comp rx=0 tx=0).stop 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.171+0000 7f3833e00640 1 -- 172.21.15.163:0/3644296568 reap_dead start 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.172+0000 7f383a400640 1 -- 172.21.15.163:0/3644296568 shutdown_connections 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.172+0000 7f383a400640 1 -- 172.21.15.163:0/3644296568 >> 172.21.15.163:0/3644296568 conn(0x7f383406bf40 msgr2=0x7f383406c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.172+0000 7f383a400640 1 -- 172.21.15.163:0/3644296568 shutdown_connections 2024-07-26T06:16:50.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.172+0000 7f383a400640 1 -- 172.21.15.163:0/3644296568 wait complete. 2024-07-26T06:16:50.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.173+0000 7f383a400640 1 Processor -- start 2024-07-26T06:16:50.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.173+0000 7f383a400640 1 -- start start 2024-07-26T06:16:50.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.174+0000 7f383a400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3818003fd0 0x7f383413d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:50.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.174+0000 7f383a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 0x7f38341566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:50.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.174+0000 7f383a400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38341e5ea0 0x7f38341e6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:50.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.174+0000 7f383a400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3834074610 con 0x7f383413dad0 2024-07-26T06:16:50.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.174+0000 7f383a400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3834074490 con 0x7f38341e5ea0 2024-07-26T06:16:50.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.174+0000 7f383a400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3834074790 con 0x7f3818003fd0 2024-07-26T06:16:50.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 0x7f38341566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:50.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 0x7f38341566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35074/0 (socket says 172.21.15.163:35074) 2024-07-26T06:16:50.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 -- 172.21.15.163:0/2943405419 learned_addr learned my addr 172.21.15.163:0/2943405419 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:50.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3833400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3818003fd0 0x7f383413d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:50.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 -- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3818003fd0 msgr2=0x7f383413d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:50.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3818003fd0 0x7f383413d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 -- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38341e5ea0 msgr2=0x7f38341e6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:50.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38341e5ea0 0x7f38341e6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 -- 172.21.15.163:0/2943405419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38341580b0 con 0x7f383413dad0 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.175+0000 7f3832a00640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 0x7f38341566a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f381c02b8c0 tx=0x7f381c033dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.176+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c043070 con 0x7f383413dad0 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.176+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f381c003c80 con 0x7f383413dad0 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.176+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c038040 con 0x7f383413dad0 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.176+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3834157010 con 0x7f383413dad0 2024-07-26T06:16:50.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.176+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38341575d0 con 0x7f383413dad0 2024-07-26T06:16:50.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.177+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3834069810 con 0x7f383413dad0 2024-07-26T06:16:50.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.178+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f381c046070 con 0x7f383413dad0 2024-07-26T06:16:50.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.179+0000 7f382b400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381007a8c0 0x7f381007cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:50.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.179+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5853+0+0 (secure 0 0 0) 0x7f381c0b86c0 con 0x7f383413dad0 2024-07-26T06:16:50.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.179+0000 7f3833400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381007a8c0 0x7f381007cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:50.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.181+0000 7f3833400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381007a8c0 0x7f381007cd80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f382c006970 tx=0x7f382c0015a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:50.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.183+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f381c081590 con 0x7f383413dad0 2024-07-26T06:16:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.436+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f383415b610 con 0x7f383413dad0 2024-07-26T06:16:50.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.944+0000 7f382b400640 1 -- 172.21.15.163:0/2943405419 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f381c085890 con 0x7f383413dad0 2024-07-26T06:16:50.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381007a8c0 msgr2=0x7f381007cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:50.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381007a8c0 0x7f381007cd80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f382c006970 tx=0x7f382c0015a0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 msgr2=0x7f38341566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 0x7f38341566a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f381c02b8c0 tx=0x7f381c033dd0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 shutdown_connections 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381007a8c0 0x7f381007cd80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38341e5ea0 0x7f38341e6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383413dad0 0x7f38341566a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 --2- 172.21.15.163:0/2943405419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3818003fd0 0x7f383413d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:50.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 >> 172.21.15.163:0/2943405419 conn(0x7f383406bf40 msgr2=0x7f38340b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:50.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 shutdown_connections 2024-07-26T06:16:50.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:50.948+0000 7f383a400640 1 -- 172.21.15.163:0/2943405419 wait complete. 2024-07-26T06:16:50.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:16:51.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 -- 172.21.15.163:0/3410077206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 msgr2=0x7f2b3806c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 --2- 172.21.15.163:0/3410077206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b3806c320 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2b38070bc0 tx=0x7f2b3002a8f0 comp rx=0 tx=0).stop 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 -- 172.21.15.163:0/3410077206 shutdown_connections 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 --2- 172.21.15.163:0/3410077206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b3806c320 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 --2- 172.21.15.163:0/3410077206 >> v2:172.21.15.164:3300/0 conn(0x7f2b380cb7e0 0x7f2b3815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 -- 172.21.15.163:0/3410077206 >> 172.21.15.163:0/3410077206 conn(0x7f2b3806d420 msgr2=0x7f2b3806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.181+0000 7f2b3da00640 1 -- 172.21.15.163:0/3410077206 shutdown_connections 2024-07-26T06:16:51.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.182+0000 7f2b3da00640 1 -- 172.21.15.163:0/3410077206 wait complete. 2024-07-26T06:16:51.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.183+0000 7f2b3da00640 1 Processor -- start 2024-07-26T06:16:51.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.184+0000 7f2b3da00640 1 -- start start 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b3da00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b381514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b3da00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b38071cf0 0x7f2b38145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b3da00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b380cb7e0 0x7f2b3815d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b3da00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b380736b0 con 0x7f2b3806bf40 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b3da00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2b38073530 con 0x7f2b38071cf0 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b3da00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2b38073830 con 0x7f2b380cb7e0 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b36a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b381514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b36a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b381514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35086/0 (socket says 172.21.15.163:35086) 2024-07-26T06:16:51.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.185+0000 7f2b36a00640 1 -- 172.21.15.163:0/1846585853 learned_addr learned my addr 172.21.15.163:0/1846585853 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:51.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b37400640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b380cb7e0 0x7f2b3815d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b36a00640 1 -- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b380cb7e0 msgr2=0x7f2b3815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b2fe00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b38071cf0 0x7f2b38145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b36a00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b380cb7e0 0x7f2b3815d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b36a00640 1 -- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b38071cf0 msgr2=0x7f2b38145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b36a00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b38071cf0 0x7f2b38145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b36a00640 1 -- 172.21.15.163:0/1846585853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b38151d20 con 0x7f2b3806bf40 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b2fe00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b38071cf0 0x7f2b38145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b36a00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b381514d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800b410 tx=0x7f2b2800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28015020 con 0x7f2b3806bf40 2024-07-26T06:16:51.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b28011940 con 0x7f2b3806bf40 2024-07-26T06:16:51.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28011d90 con 0x7f2b3806bf40 2024-07-26T06:16:51.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b38152000 con 0x7f2b3806bf40 2024-07-26T06:16:51.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.186+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b38151a10 con 0x7f2b3806bf40 2024-07-26T06:16:51.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.187+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b380b0d80 con 0x7f2b3806bf40 2024-07-26T06:16:51.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.188+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f2b28018020 con 0x7f2b3806bf40 2024-07-26T06:16:51.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.189+0000 7f2b2ea00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2b140764b0 0x7f2b14078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:51.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.190+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5853+0+0 (secure 0 0 0) 0x7f2b28097a90 con 0x7f2b3806bf40 2024-07-26T06:16:51.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.190+0000 7f2b2fe00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2b140764b0 0x7f2b14078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:51.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.191+0000 7f2b2fe00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2b140764b0 0x7f2b14078970 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2b20007fd0 tx=0x7f2b20009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:51.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.193+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2b380b0d80 con 0x7f2b3806bf40 2024-07-26T06:16:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:51.383+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2b38074590 con 0x7f2b3806bf40 2024-07-26T06:16:52.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.152+0000 7f2b2ea00640 1 -- 172.21.15.163:0/1846585853 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) ==== 181+0+0 (secure 0 0 0) 0x7f2b2801f020 con 0x7f2b3806bf40 2024-07-26T06:16:52.154 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:16:52.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2b140764b0 msgr2=0x7f2b14078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:52.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2b140764b0 0x7f2b14078970 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2b20007fd0 tx=0x7f2b20009040 comp rx=0 tx=0).stop 2024-07-26T06:16:52.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 msgr2=0x7f2b381514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b381514d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800b410 tx=0x7f2b2800b8d0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 shutdown_connections 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2b140764b0 0x7f2b14078970 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2b20007fd0 tx=0x7f2b20009040 comp rx=0 tx=0).stop 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b380cb7e0 0x7f2b3815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b38071cf0 0x7f2b38145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 --2- 172.21.15.163:0/1846585853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b3806bf40 0x7f2b381514d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.157+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 >> 172.21.15.163:0/1846585853 conn(0x7f2b3806d420 msgr2=0x7f2b3806d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:52.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.158+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 shutdown_connections 2024-07-26T06:16:52.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.158+0000 7f2b3da00640 1 -- 172.21.15.163:0/1846585853 wait complete. 2024-07-26T06:16:52.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:16:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 -- 172.21.15.163:0/3653377336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c15e440 msgr2=0x7f2b0c0adc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3653377336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c15e440 0x7f2b0c0adc80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2af4005fd0 tx=0x7f2af402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 -- 172.21.15.163:0/3653377336 shutdown_connections 2024-07-26T06:16:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3653377336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c15e440 0x7f2b0c0adc80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3653377336 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b0c15db20 0x7f2b0c15df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 -- 172.21.15.163:0/3653377336 >> 172.21.15.163:0/3653377336 conn(0x7f2b0c0b1c40 msgr2=0x7f2b0c0b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 -- 172.21.15.163:0/3653377336 shutdown_connections 2024-07-26T06:16:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.392+0000 7f2b10a00640 1 -- 172.21.15.163:0/3653377336 wait complete. 2024-07-26T06:16:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.394+0000 7f2b10a00640 1 Processor -- start 2024-07-26T06:16:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.394+0000 7f2b10a00640 1 -- start start 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b10a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 0x7f2b0c148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b10a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b0c15db20 0x7f2b0c1f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b10a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b0c15e440 0x7f2b0c1f5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b10a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c06ce70 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b10a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c06ccf0 con 0x7f2b0c15db20 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b10a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0c06cff0 con 0x7f2b0c15e440 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b09e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 0x7f2b0c148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b09e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 0x7f2b0c148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35096/0 (socket says 172.21.15.163:35096) 2024-07-26T06:16:52.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b09e00640 1 -- 172.21.15.163:0/3772384938 learned_addr learned my addr 172.21.15.163:0/3772384938 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:52.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b0a800640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b0c15e440 0x7f2b0c1f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:52.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.395+0000 7f2b01400640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b0c15db20 0x7f2b0c1f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b09e00640 1 -- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b0c15e440 msgr2=0x7f2b0c1f5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b09e00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b0c15e440 0x7f2b0c1f5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b09e00640 1 -- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b0c15db20 msgr2=0x7f2b0c1f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b09e00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b0c15db20 0x7f2b0c1f4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b09e00640 1 -- 172.21.15.163:0/3772384938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c1f5c80 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b09e00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 0x7f2b0c148df0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00d950 tx=0x7f2afc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc017070 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2afc009d20 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0106c0 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0c1f5f00 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.396+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0c1f64c0 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.398+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0c06d510 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.399+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f2afc010860 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.399+0000 7f2b02a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2ae8076480 0x7f2ae8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:52.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.400+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5415+0+0 (secure 0 0 0) 0x7f2afc097e30 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.400+0000 7f2b01400640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2ae8076480 0x7f2ae8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:52.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.401+0000 7f2b01400640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2ae8076480 0x7f2ae8078940 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2af4033000 tx=0x7f2af4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:52.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.404+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2b0c1f5c80 con 0x7f2b0c0ae1c0 2024-07-26T06:16:52.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:52.609+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2b0c0732a0 con 0x7f2b0c0ae1c0 2024-07-26T06:16:53.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.164+0000 7f2b02a00640 1 -- 172.21.15.163:0/3772384938 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v50) ==== 169+0+0 (secure 0 0 0) 0x7f2afc061510 con 0x7f2b0c0ae1c0 2024-07-26T06:16:53.166 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:16:53.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.171+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2ae8076480 msgr2=0x7f2ae8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.171+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2ae8076480 0x7f2ae8078940 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2af4033000 tx=0x7f2af4004730 comp rx=0 tx=0).stop 2024-07-26T06:16:53.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.171+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 msgr2=0x7f2b0c148df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.171+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 0x7f2b0c148df0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00d950 tx=0x7f2afc00de10 comp rx=0 tx=0).stop 2024-07-26T06:16:53.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 shutdown_connections 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2ae8076480 0x7f2ae8078940 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b0c15e440 0x7f2b0c1f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b0c15db20 0x7f2b0c1f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 --2- 172.21.15.163:0/3772384938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b0c0ae1c0 0x7f2b0c148df0 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00d950 tx=0x7f2afc00de10 comp rx=0 tx=0).stop 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 >> 172.21.15.163:0/3772384938 conn(0x7f2b0c0b1c40 msgr2=0x7f2b0c0ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 shutdown_connections 2024-07-26T06:16:53.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.172+0000 7f2b10a00640 1 -- 172.21.15.163:0/3772384938 wait complete. 2024-07-26T06:16:53.199 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 -- 172.21.15.163:0/2284910570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec0b4420 msgr2=0x7f22ec0b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 --2- 172.21.15.163:0/2284910570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec0b4420 0x7f22ec0b2130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0b1bc0 tx=0x7f22e402aa30 comp rx=0 tx=0).stop 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 -- 172.21.15.163:0/2284910570 shutdown_connections 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 --2- 172.21.15.163:0/2284910570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec0b4420 0x7f22ec0b2130 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 --2- 172.21.15.163:0/2284910570 >> v2:172.21.15.164:3300/0 conn(0x7f22ec0cbad0 0x7f22ec159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 -- 172.21.15.163:0/2284910570 >> 172.21.15.163:0/2284910570 conn(0x7f22ec0b0e30 msgr2=0x7f22ec0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 -- 172.21.15.163:0/2284910570 shutdown_connections 2024-07-26T06:16:53.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.387+0000 7f22f0c00640 1 -- 172.21.15.163:0/2284910570 wait complete. 2024-07-26T06:16:53.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.388+0000 7f22f0c00640 1 Processor -- start 2024-07-26T06:16:53.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.389+0000 7f22f0c00640 1 -- start start 2024-07-26T06:16:53.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.389+0000 7f22f0c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ec0b4420 0x7f22ec1f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22f0c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 0x7f22ec150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22f0c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22ec073bd0 0x7f22ec072270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22f0c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec06f2f0 con 0x7f22ec159370 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22f0c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec0cd210 con 0x7f22ec0b4420 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22f0c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f22ec06f470 con 0x7f22ec073bd0 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 0x7f22ec150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 0x7f22ec150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35112/0 (socket says 172.21.15.163:35112) 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 -- 172.21.15.163:0/3979743570 learned_addr learned my addr 172.21.15.163:0/3979743570 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 -- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22ec073bd0 msgr2=0x7f22ec072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22ec073bd0 0x7f22ec072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 -- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ec0b4420 msgr2=0x7f22ec1f4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eb400640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ec0b4420 0x7f22ec1f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ec0b4420 0x7f22ec1f4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.390+0000 7f22eaa00640 1 -- 172.21.15.163:0/3979743570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ec074080 con 0x7f22ec159370 2024-07-26T06:16:53.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.391+0000 7f22eaa00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 0x7f22ec150ca0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f22d800b4a0 tx=0x7f22d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.391+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8014070 con 0x7f22ec159370 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.391+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22d8010700 con 0x7f22ec159370 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.391+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8010b50 con 0x7f22ec159370 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.391+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22ec0742f0 con 0x7f22ec159370 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.391+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22ec1f4130 con 0x7f22ec159370 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.392+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22ec0b2130 con 0x7f22ec159370 2024-07-26T06:16:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.393+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f22d8017070 con 0x7f22ec159370 2024-07-26T06:16:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.394+0000 7f22e3400640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f22cc076480 0x7f22cc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.394+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7f22d8097230 con 0x7f22ec159370 2024-07-26T06:16:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.395+0000 7f22eb400640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f22cc076480 0x7f22cc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.396+0000 7f22eb400640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f22cc076480 0x7f22cc078940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22dc003d90 tx=0x7f22dc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:53.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.399+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22d8060c00 con 0x7f22ec159370 2024-07-26T06:16:53.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.608+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f22ec0af0e0 con 0x7f22ec159370 2024-07-26T06:16:53.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.615+0000 7f22e3400640 1 -- 172.21.15.163:0/3979743570 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v31) ==== 71+0+0 (secure 0 0 0) 0x7f22d8060500 con 0x7f22ec159370 2024-07-26T06:16:53.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f22cc076480 msgr2=0x7f22cc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f22cc076480 0x7f22cc078940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22dc003d90 tx=0x7f22dc009040 comp rx=0 tx=0).stop 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 msgr2=0x7f22ec150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 0x7f22ec150ca0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f22d800b4a0 tx=0x7f22d800b960 comp rx=0 tx=0).stop 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 shutdown_connections 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f22cc076480 0x7f22cc078940 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22ec073bd0 0x7f22ec072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22ec159370 0x7f22ec150ca0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 --2- 172.21.15.163:0/3979743570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ec0b4420 0x7f22ec1f4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.618+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 >> 172.21.15.163:0/3979743570 conn(0x7f22ec0b0e30 msgr2=0x7f22ec151d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.619+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 shutdown_connections 2024-07-26T06:16:53.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.619+0000 7f22f0c00640 1 -- 172.21.15.163:0/3979743570 wait complete. 2024-07-26T06:16:53.637 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-07-26T06:16:53.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 -- 172.21.15.163:0/2098554154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 msgr2=0x7fd6f000b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 --2- 172.21.15.163:0/2098554154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd6f000b9e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd70400ad20 tx=0x7fd6f0005830 comp rx=0 tx=0).stop 2024-07-26T06:16:53.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 -- 172.21.15.163:0/2098554154 shutdown_connections 2024-07-26T06:16:53.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 --2- 172.21.15.163:0/2098554154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd6f000b9e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 -- 172.21.15.163:0/2098554154 >> 172.21.15.163:0/2098554154 conn(0x7fd70c06d420 msgr2=0x7fd70c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:53.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 -- 172.21.15.163:0/2098554154 shutdown_connections 2024-07-26T06:16:53.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.851+0000 7fd713c00640 1 -- 172.21.15.163:0/2098554154 wait complete. 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.853+0000 7fd713c00640 1 Processor -- start 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.853+0000 7fd713c00640 1 -- start start 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd713c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd70c150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd713c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70c0cdef0 0x7fd70c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd713c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70c153ac0 0x7fd70c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd713c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c06bf40 con 0x7fd6f0005370 2024-07-26T06:16:53.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd713c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c15f040 con 0x7fd70c153ac0 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd713c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd70c06c0c0 con 0x7fd70c0cdef0 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd70c150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd70c150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35124/0 (socket says 172.21.15.163:35124) 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 -- 172.21.15.163:0/259506557 learned_addr learned my addr 172.21.15.163:0/259506557 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 -- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70c0cdef0 msgr2=0x7fd70c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70c0cdef0 0x7fd70c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 -- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70c153ac0 msgr2=0x7fd70c153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:53.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70c153ac0 0x7fd70c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.854+0000 7fd711200640 1 -- 172.21.15.163:0/259506557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c154480 con 0x7fd6f0005370 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.855+0000 7fd711200640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd70c150a70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc001960 tx=0x7fd6fc00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.855+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc019070 con 0x7fd6f0005370 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.855+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6fc0050d0 con 0x7fd6f0005370 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.855+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c1515b0 con 0x7fd6f0005370 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.855+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc00b660 con 0x7fd6f0005370 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.855+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c1f7440 con 0x7fd6f0005370 2024-07-26T06:16:53.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.856+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd70c0ce890 con 0x7fd6f0005370 2024-07-26T06:16:53.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.857+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fd6fc01c070 con 0x7fd6f0005370 2024-07-26T06:16:53.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.858+0000 7fd708c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd6ec0764b0 0x7fd6ec078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:53.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.858+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7fd6fc096fd0 con 0x7fd6f0005370 2024-07-26T06:16:53.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.859+0000 7fd70be00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd6ec0764b0 0x7fd6ec078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:53.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.860+0000 7fd70be00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd6ec0764b0 0x7fd6ec078970 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0ceb80 tx=0x7fd6f80069c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:53.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:53.862+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd6fc060910 con 0x7fd6f0005370 2024-07-26T06:16:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.146+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7fd70c15f1c0 con 0x7fd6f0005370 2024-07-26T06:16:54.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.168+0000 7fd708c00640 1 -- 172.21.15.163:0/259506557 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v176) ==== 140+0+0 (secure 0 0 0) 0x7fd6fc060210 con 0x7fd6f0005370 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd6ec0764b0 msgr2=0x7fd6ec078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd6ec0764b0 0x7fd6ec078970 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0ceb80 tx=0x7fd6f80069c0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 msgr2=0x7fd70c150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd70c150a70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc001960 tx=0x7fd6fc00db80 comp rx=0 tx=0).stop 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 shutdown_connections 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd6ec0764b0 0x7fd6ec078970 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70c153ac0 0x7fd70c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70c0cdef0 0x7fd70c0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 --2- 172.21.15.163:0/259506557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6f0005370 0x7fd70c150a70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 >> 172.21.15.163:0/259506557 conn(0x7fd70c06d420 msgr2=0x7fd70c0b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:54.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 shutdown_connections 2024-07-26T06:16:54.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.171+0000 7fd713c00640 1 -- 172.21.15.163:0/259506557 wait complete. 2024-07-26T06:16:54.190 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-07-26T06:16:54.191 INFO:tasks.cephfs_test_runner:Starting test: test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate handcrafted .meta file on legacy subvol root doesn't break the system 2024-07-26T06:16:54.191 INFO:tasks.cephfs_test_runner:test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:16:54.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:16:54.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 -- 172.21.15.163:0/784214525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 msgr2=0x7f20c80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 --2- 172.21.15.163:0/784214525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80ce2f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f20b8005fd0 tx=0x7f20b802a540 comp rx=0 tx=0).stop 2024-07-26T06:16:54.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 -- 172.21.15.163:0/784214525 shutdown_connections 2024-07-26T06:16:54.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 --2- 172.21.15.163:0/784214525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80ce2f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 -- 172.21.15.163:0/784214525 >> 172.21.15.163:0/784214525 conn(0x7f20c80cfda0 msgr2=0x7f20c80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:54.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 -- 172.21.15.163:0/784214525 shutdown_connections 2024-07-26T06:16:54.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.396+0000 7f20cea00640 1 -- 172.21.15.163:0/784214525 wait complete. 2024-07-26T06:16:54.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.398+0000 7f20cea00640 1 Processor -- start 2024-07-26T06:16:54.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.398+0000 7f20cea00640 1 -- start start 2024-07-26T06:16:54.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20cea00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c80b1e80 0x7f20c80cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20cea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c80ce830 0x7f20c81f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20cea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20cea00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20c80b6f60 con 0x7f20c815d5a0 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20cea00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20c80b6de0 con 0x7f20c80ce830 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20cea00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20c80b54b0 con 0x7f20c80b1e80 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20ce000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20ce000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35144/0 (socket says 172.21.15.163:35144) 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.399+0000 7f20ce000640 1 -- 172.21.15.163:0/642136285 learned_addr learned my addr 172.21.15.163:0/642136285 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20cd600640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c80b1e80 0x7f20c80cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ce000640 1 -- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c80b1e80 msgr2=0x7f20c80cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ce000640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c80b1e80 0x7f20c80cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ce000640 1 -- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c80ce830 msgr2=0x7f20c81f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ccc00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c80ce830 0x7f20c81f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ce000640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c80ce830 0x7f20c81f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ce000640 1 -- 172.21.15.163:0/642136285 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c80ccb20 con 0x7f20c815d5a0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20ce000640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80c8c70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c7b0 tx=0x7f20c000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0004b60 con 0x7f20c815d5a0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20c0004df0 con 0x7f20c815d5a0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c815fd80 con 0x7f20c815d5a0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.400+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c000aa90 con 0x7f20c815d5a0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.401+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c8160340 con 0x7f20c815d5a0 2024-07-26T06:16:54.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.402+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c80cf860 con 0x7f20c815d5a0 2024-07-26T06:16:54.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.403+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f20c000ac30 con 0x7f20c815d5a0 2024-07-26T06:16:54.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.404+0000 7f20c5600640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f20a8076480 0x7f20a8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.404+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7f20c0097ab0 con 0x7f20c815d5a0 2024-07-26T06:16:54.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.404+0000 7f20cd600640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f20a8076480 0x7f20a8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.406+0000 7f20cd600640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f20a8076480 0x7f20a8078940 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f20b8033000 tx=0x7f20b8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:54.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.408+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f20c80cf860 con 0x7f20c815d5a0 2024-07-26T06:16:54.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.619+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20c80ce2f0 con 0x7f20c815d5a0 2024-07-26T06:16:54.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.620+0000 7f20c5600640 1 -- 172.21.15.163:0/642136285 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+13046 (secure 0 0 0) 0x7f20c0020070 con 0x7f20c815d5a0 2024-07-26T06:16:54.621 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:54.621 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":50,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:16:53.153785+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6835/1618066632":"2024-07-27T06:16:49.936998+0000","172.21.15.164:6834/1618066632":"2024-07-27T06:16:49.936998+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:16:54.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f20a8076480 msgr2=0x7f20a8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f20a8076480 0x7f20a8078940 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f20b8033000 tx=0x7f20b8004730 comp rx=0 tx=0).stop 2024-07-26T06:16:54.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 msgr2=0x7f20c80c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80c8c70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c7b0 tx=0x7f20c000cc70 comp rx=0 tx=0).stop 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 shutdown_connections 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f20a8076480 0x7f20a8078940 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c815d5a0 0x7f20c80c8c70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c80ce830 0x7f20c81f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 --2- 172.21.15.163:0/642136285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c80b1e80 0x7f20c80cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 >> 172.21.15.163:0/642136285 conn(0x7f20c80cfda0 msgr2=0x7f20c815a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 shutdown_connections 2024-07-26T06:16:54.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.623+0000 7f20cea00640 1 -- 172.21.15.163:0/642136285 wait complete. 2024-07-26T06:16:54.642 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:13:21.677101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:16:54.643 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T06:16:54.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 -- 172.21.15.163:0/2313298259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 msgr2=0x7fdce80b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 --2- 172.21.15.163:0/2313298259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce80b5800 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc005fd0 tx=0x7fdcdc02aa80 comp rx=0 tx=0).stop 2024-07-26T06:16:54.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 -- 172.21.15.163:0/2313298259 shutdown_connections 2024-07-26T06:16:54.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 --2- 172.21.15.163:0/2313298259 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdce80b6720 0x7fdce80b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 --2- 172.21.15.163:0/2313298259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce80b5800 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 -- 172.21.15.163:0/2313298259 >> 172.21.15.163:0/2313298259 conn(0x7fdce80d36b0 msgr2=0x7fdce80d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:54.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 -- 172.21.15.163:0/2313298259 shutdown_connections 2024-07-26T06:16:54.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.857+0000 7fdcf0200640 1 -- 172.21.15.163:0/2313298259 wait complete. 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.859+0000 7fdcf0200640 1 Processor -- start 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.859+0000 7fdcf0200640 1 -- start start 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcf0200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcf0200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdce80b6720 0x7fdce8160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcf0200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdce815d650 0x7fdce80ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcf0200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdce80b7250 con 0x7fdce80b53b0 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcf0200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdce80b70d0 con 0x7fdce815d650 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcf0200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdce80b73d0 con 0x7fdce80b6720 2024-07-26T06:16:54.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdced800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdced800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce815dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41366/0 (socket says 172.21.15.163:41366) 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdced800640 1 -- 172.21.15.163:0/1699244747 learned_addr learned my addr 172.21.15.163:0/1699244747 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.860+0000 7fdcece00640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdce80b6720 0x7fdce8160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdcee200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdce815d650 0x7fdce80ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdced800640 1 -- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdce80b6720 msgr2=0x7fdce8160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdced800640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdce80b6720 0x7fdce8160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdced800640 1 -- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdce815d650 msgr2=0x7fdce80ab720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdced800640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdce815d650 0x7fdce80ab720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdced800640 1 -- 172.21.15.163:0/1699244747 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce80ac0e0 con 0x7fdce80b53b0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdcee200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdce815d650 0x7fdce80ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdced800640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce815dba0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800b3f0 tx=0x7fdcd800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8015020 con 0x7fdce80b53b0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcd8011920 con 0x7fdce80b53b0 2024-07-26T06:16:54.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce80ac6a0 con 0x7fdce80b53b0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8011d70 con 0x7fdce80b53b0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.861+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce8159a30 con 0x7fdce80b53b0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.863+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce80ca160 con 0x7fdce80b53b0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.864+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fdcd8018020 con 0x7fdce80b53b0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.865+0000 7fdce5600640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdccc076480 0x7fdccc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.865+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7fdcd80974a0 con 0x7fdce80b53b0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.866+0000 7fdcece00640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdccc076480 0x7fdccc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.867+0000 7fdcece00640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdccc076480 0x7fdccc078940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc035000 tx=0x7fdcdc039040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:54.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:54.869+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdce80ca160 con 0x7fdce80b53b0 2024-07-26T06:16:55.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.073+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdce80b7880 con 0x7fdce80b53b0 2024-07-26T06:16:55.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.074+0000 7fdce5600640 1 -- 172.21.15.163:0/1699244747 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7fdcd8067e20 con 0x7fdce80b53b0 2024-07-26T06:16:55.074 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdccc076480 msgr2=0x7fdccc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdccc076480 0x7fdccc078940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc035000 tx=0x7fdcdc039040 comp rx=0 tx=0).stop 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 msgr2=0x7fdce815dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce815dba0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800b3f0 tx=0x7fdcd800b8b0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 shutdown_connections 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdccc076480 0x7fdccc078940 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdce815d650 0x7fdce80ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdce80b6720 0x7fdce8160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 --2- 172.21.15.163:0/1699244747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdce80b53b0 0x7fdce815dba0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 >> 172.21.15.163:0/1699244747 conn(0x7fdce80d36b0 msgr2=0x7fdce80b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:55.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 shutdown_connections 2024-07-26T06:16:55.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.077+0000 7fdcf0200640 1 -- 172.21.15.163:0/1699244747 wait complete. 2024-07-26T06:16:55.096 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-07-26T06:16:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 -- 172.21.15.163:0/158603106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 msgr2=0x7f98a80b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 --2- 172.21.15.163:0/158603106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80b6bd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009830 tx=0x7f98a002a3f0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 -- 172.21.15.163:0/158603106 shutdown_connections 2024-07-26T06:16:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 --2- 172.21.15.163:0/158603106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80b6bd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 -- 172.21.15.163:0/158603106 >> 172.21.15.163:0/158603106 conn(0x7f98a80cfda0 msgr2=0x7f98a80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 -- 172.21.15.163:0/158603106 shutdown_connections 2024-07-26T06:16:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.295+0000 7f98ad400640 1 -- 172.21.15.163:0/158603106 wait complete. 2024-07-26T06:16:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.297+0000 7f98ad400640 1 Processor -- start 2024-07-26T06:16:55.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.297+0000 7f98ad400640 1 -- start start 2024-07-26T06:16:55.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98ad400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b1e80 0x7f98a81518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98ad400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98ad400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98a815d5a0 0x7f98a81f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98ad400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f98a80b5130 con 0x7f98a80b23d0 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98ad400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f98a80b4fb0 con 0x7f98a815d5a0 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98ad400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f98a80b52b0 con 0x7f98a80b1e80 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98a7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98a7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41376/0 (socket says 172.21.15.163:41376) 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.298+0000 7f98a7400640 1 -- 172.21.15.163:0/2549908819 learned_addr learned my addr 172.21.15.163:0/2549908819 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:55.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7e00640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b1e80 0x7f98a81518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98aca00640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98a815d5a0 0x7f98a81f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7400640 1 -- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b1e80 msgr2=0x7f98a81518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b1e80 0x7f98a81518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7400640 1 -- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98a815d5a0 msgr2=0x7f98a81f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98a815d5a0 0x7f98a81f3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7400640 1 -- 172.21.15.163:0/2549908819 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a81f3960 con 0x7f98a80b23d0 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7e00640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b1e80 0x7f98a81518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98a7400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80c93d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f989000c450 tx=0x7f989000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890016020 con 0x7f98a80b23d0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9890012920 con 0x7f98a80b23d0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890012d70 con 0x7f98a80b23d0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.299+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a8151ea0 con 0x7f98a80b23d0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.300+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a8152460 con 0x7f98a80b23d0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.301+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a80b57b0 con 0x7f98a80b23d0 2024-07-26T06:16:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.302+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f9890019020 con 0x7f98a80b23d0 2024-07-26T06:16:55.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.303+0000 7f989fe00640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f98840764b0 0x7f9884078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:55.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.303+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7f98900974a0 con 0x7f98a80b23d0 2024-07-26T06:16:55.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.303+0000 7f98a7e00640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f98840764b0 0x7f9884078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.305+0000 7f98a7e00640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f98840764b0 0x7f9884078970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9898007e90 tx=0x7f98980073a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:55.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.307+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9890020030 con 0x7f98a80b23d0 2024-07-26T06:16:55.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:55.591+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f98a8069880 con 0x7f98a80b23d0 2024-07-26T06:16:56.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.181+0000 7f989fe00640 1 -- 172.21.15.163:0/2549908819 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v178) ==== 164+0+0 (secure 0 0 0) 0x7f98a81f3960 con 0x7f98a80b23d0 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f98840764b0 msgr2=0x7f9884078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f98840764b0 0x7f9884078970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9898007e90 tx=0x7f98980073a0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 msgr2=0x7f98a80c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80c93d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f989000c450 tx=0x7f989000c910 comp rx=0 tx=0).stop 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 shutdown_connections 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f98840764b0 0x7f9884078970 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98a815d5a0 0x7f98a81f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98a80b23d0 0x7f98a80c93d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 --2- 172.21.15.163:0/2549908819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b1e80 0x7f98a81518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 >> 172.21.15.163:0/2549908819 conn(0x7f98a80cfda0 msgr2=0x7f98a815aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:56.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 shutdown_connections 2024-07-26T06:16:56.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.185+0000 7f98ad400640 1 -- 172.21.15.163:0/2549908819 wait complete. 2024-07-26T06:16:56.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.399+0000 7f2821000640 1 -- 172.21.15.163:0/431097914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c071a70 msgr2=0x7f281c0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 --2- 172.21.15.163:0/431097914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c071a70 0x7f281c0b3710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2814005fd0 tx=0x7f281402a840 comp rx=0 tx=0).stop 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 -- 172.21.15.163:0/431097914 shutdown_connections 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 --2- 172.21.15.163:0/431097914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c071a70 0x7f281c0b3710 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 --2- 172.21.15.163:0/431097914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f281c0ce4a0 0x7f281c071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 -- 172.21.15.163:0/431097914 >> 172.21.15.163:0/431097914 conn(0x7f281c0b0e30 msgr2=0x7f281c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 -- 172.21.15.163:0/431097914 shutdown_connections 2024-07-26T06:16:56.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.400+0000 7f2821000640 1 -- 172.21.15.163:0/431097914 wait complete. 2024-07-26T06:16:56.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.401+0000 7f2821000640 1 Processor -- start 2024-07-26T06:16:56.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.402+0000 7f2821000640 1 -- start start 2024-07-26T06:16:56.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.402+0000 7f2821000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f281c071a70 0x7f281c148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.402+0000 7f2821000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f281c0ce4a0 0x7f281c15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f2821000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 0x7f281c15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f2821000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f281c0b4020 con 0x7f281c1556f0 2024-07-26T06:16:56.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f2821000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f281c0b3ea0 con 0x7f281c0ce4a0 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f2821000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f281c0b41a0 con 0x7f281c071a70 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 0x7f281c15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 0x7f281c15a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41392/0 (socket says 172.21.15.163:41392) 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 -- 172.21.15.163:0/2497870673 learned_addr learned my addr 172.21.15.163:0/2497870673 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f2813e00640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f281c0ce4a0 0x7f281c15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281b400640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f281c071a70 0x7f281c148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 -- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f281c071a70 msgr2=0x7f281c148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f281c071a70 0x7f281c148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 -- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f281c0ce4a0 msgr2=0x7f281c15a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f281c0ce4a0 0x7f281c15a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f281be00640 1 -- 172.21.15.163:0/2497870673 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c0cbd60 con 0x7f281c1556f0 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.403+0000 7f2813e00640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f281c0ce4a0 0x7f281c15a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:56.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.404+0000 7f281be00640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 0x7f281c15a590 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2814034040 tx=0x7f2814031610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:56.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.404+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2814042030 con 0x7f281c1556f0 2024-07-26T06:16:56.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.404+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2814031a10 con 0x7f281c1556f0 2024-07-26T06:16:56.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.404+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2814031e80 con 0x7f281c1556f0 2024-07-26T06:16:56.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.404+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281c0cc040 con 0x7f281c1556f0 2024-07-26T06:16:56.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.404+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f281c0cc600 con 0x7f281c1556f0 2024-07-26T06:16:56.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.405+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f281c0b4730 con 0x7f281c1556f0 2024-07-26T06:16:56.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.406+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f2814045070 con 0x7f281c1556f0 2024-07-26T06:16:56.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.407+0000 7f2813400640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27f8076480 0x7f27f8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.407+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7f28140b88b0 con 0x7f281c1556f0 2024-07-26T06:16:56.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.408+0000 7f281b400640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27f8076480 0x7f27f8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.409+0000 7f281b400640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27f8076480 0x7f27f8078940 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f280c001eb0 tx=0x7f280c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.412+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f281404c070 con 0x7f281c1556f0 2024-07-26T06:16:56.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:16:56.617 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.617+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f281c15aec0 con 0x7f281c1556f0 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.618+0000 7f2813400640 1 -- 172.21.15.163:0/2497870673 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+1998 (secure 0 0 0) 0x7f28140821f0 con 0x7f281c1556f0 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stdout:mon advanced mon_op_complaint_time 120 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stdout:mon dev mon_warn_on_pool_no_app false 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stdout:osd advanced osd_op_complaint_time 180.000000 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 5439.226762 2024-07-26T06:16:56.618 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3901.149445 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5414.054049 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4219.540367 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4287.026771 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4403.390021 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3932.844298 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4626.659828 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds 20/20 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds_balancer 20/20 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_ms 1/1 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_frag true 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_scatterstat true 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced mds_op_complaint_time 180.000000 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_verify_scatter true 2024-07-26T06:16:56.619 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced osd_op_complaint_time 180.000000 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_mon_op_timeout 900 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_osd_op_timeout 900 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced client_mount_timeout 600 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_client 20/20 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_ms 1/1 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_mon_op_timeout 900 2024-07-26T06:16:56.620 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_osd_op_timeout 900 2024-07-26T06:16:56.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27f8076480 msgr2=0x7f27f8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27f8076480 0x7f27f8078940 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f280c001eb0 tx=0x7f280c009040 comp rx=0 tx=0).stop 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 msgr2=0x7f281c15a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 0x7f281c15a590 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2814034040 tx=0x7f2814031610 comp rx=0 tx=0).stop 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 shutdown_connections 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f27f8076480 0x7f27f8078940 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f281c1556f0 0x7f281c15a590 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f281c0ce4a0 0x7f281c15a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 --2- 172.21.15.163:0/2497870673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f281c071a70 0x7f281c148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 >> 172.21.15.163:0/2497870673 conn(0x7f281c0b0e30 msgr2=0x7f281c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 shutdown_connections 2024-07-26T06:16:56.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.621+0000 7f2821000640 1 -- 172.21.15.163:0/2497870673 wait complete. 2024-07-26T06:16:56.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:16:56.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e70a00640 1 -- 172.21.15.163:0/3216664862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e500040d0 msgr2=0x7f2e50004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e70a00640 1 --2- 172.21.15.163:0/3216664862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e500040d0 0x7f2e50004800 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c06cf00 tx=0x7f2e50005830 comp rx=0 tx=0).stop 2024-07-26T06:16:56.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e6b200640 1 -- 172.21.15.163:0/3216664862 reap_dead start 2024-07-26T06:16:56.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e70a00640 1 -- 172.21.15.163:0/3216664862 shutdown_connections 2024-07-26T06:16:56.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e70a00640 1 -- 172.21.15.163:0/3216664862 >> 172.21.15.163:0/3216664862 conn(0x7f2e6c0b0e30 msgr2=0x7f2e6c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:56.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e70a00640 1 -- 172.21.15.163:0/3216664862 shutdown_connections 2024-07-26T06:16:56.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.884+0000 7f2e70a00640 1 -- 172.21.15.163:0/3216664862 wait complete. 2024-07-26T06:16:56.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.886+0000 7f2e70a00640 1 Processor -- start 2024-07-26T06:16:56.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.886+0000 7f2e70a00640 1 -- start start 2024-07-26T06:16:56.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e70a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e500040d0 0x7f2e6c14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e70a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e6c1f7980 0x7f2e6c1e7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e70a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 0x7f2e6c150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e70a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c071530 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e70a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c071ee0 con 0x7f2e500040d0 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e70a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6c0716b0 con 0x7f2e6c1f7980 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 0x7f2e6c150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 0x7f2e6c150da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41418/0 (socket says 172.21.15.163:41418) 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 -- 172.21.15.163:0/2433729370 learned_addr learned my addr 172.21.15.163:0/2433729370 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6a800640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e500040d0 0x7f2e6c14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e63e00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e6c1f7980 0x7f2e6c1e7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 -- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e6c1f7980 msgr2=0x7f2e6c1e7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e6c1f7980 0x7f2e6c1e7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 -- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e500040d0 msgr2=0x7f2e6c14fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e500040d0 0x7f2e6c14fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:56.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e6b200640 1 -- 172.21.15.163:0/2433729370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c1e8520 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.887+0000 7f2e63e00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e6c1f7980 0x7f2e6c1e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:56.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.888+0000 7f2e6b200640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 0x7f2e6c150da0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c068de0 tx=0x7f2e6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:56.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.888+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64017070 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.888+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e6400ae40 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.888+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e640072d0 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.888+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6c14e960 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.888+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6c14ef20 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.889+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e6c0cdd90 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.890+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f2e6401a070 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.891+0000 7f2e62a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2e480764b0 0x7f2e48078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:56.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.891+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7f2e64097610 con 0x7f2e6c1e80a0 2024-07-26T06:16:56.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.892+0000 7f2e6a800640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2e480764b0 0x7f2e48078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:56.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.893+0000 7f2e6a800640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2e480764b0 0x7f2e48078970 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00dba0 tx=0x7f2e5c005f30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:56.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:56.896+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e640610a0 con 0x7f2e6c1e80a0 2024-07-26T06:16:57.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.084+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f2e6c0b34f0 con 0x7f2e6c1e80a0 2024-07-26T06:16:57.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.092+0000 7f2e62a00640 1 -- 172.21.15.163:0/2433729370 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f2e64069090 con 0x7f2e6c1e80a0 2024-07-26T06:16:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2e480764b0 msgr2=0x7f2e48078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2e480764b0 0x7f2e48078970 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00dba0 tx=0x7f2e5c005f30 comp rx=0 tx=0).stop 2024-07-26T06:16:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 msgr2=0x7f2e6c150da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 0x7f2e6c150da0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c068de0 tx=0x7f2e6400a430 comp rx=0 tx=0).stop 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 shutdown_connections 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2e480764b0 0x7f2e48078970 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e6c1e80a0 0x7f2e6c150da0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e6c1f7980 0x7f2e6c1e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 --2- 172.21.15.163:0/2433729370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e500040d0 0x7f2e6c14fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.094+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 >> 172.21.15.163:0/2433729370 conn(0x7f2e6c0b0e30 msgr2=0x7f2e6c0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.095+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 shutdown_connections 2024-07-26T06:16:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.095+0000 7f2e70a00640 1 -- 172.21.15.163:0/2433729370 wait complete. 2024-07-26T06:16:57.112 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:16:57.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:16:57.216 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:16:57.336 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:16:57.337 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:16:57.337 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:57.396 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:16:57.396 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:16:57.396 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:57.455 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:16:57.455 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:16:57.455 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:57.512 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:16:57.512 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:16:57.512 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:16:57.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:16:57.571 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-07-26T06:16:57.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.785+0000 7f2007200640 1 -- 172.21.15.163:0/1433860893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 msgr2=0x7f1fe80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.785+0000 7f2007200640 1 --2- 172.21.15.163:0/1433860893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f1fe80051c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2008068e30 tx=0x7f1fe8005830 comp rx=0 tx=0).stop 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.785+0000 7f2006800640 1 -- 172.21.15.163:0/1433860893 reap_dead start 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.786+0000 7f2007200640 1 -- 172.21.15.163:0/1433860893 shutdown_connections 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.786+0000 7f2007200640 1 -- 172.21.15.163:0/1433860893 >> 172.21.15.163:0/1433860893 conn(0x7f20080b0e30 msgr2=0x7f20080b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.786+0000 7f2007200640 1 -- 172.21.15.163:0/1433860893 shutdown_connections 2024-07-26T06:16:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.786+0000 7f2007200640 1 -- 172.21.15.163:0/1433860893 wait complete. 2024-07-26T06:16:57.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.787+0000 7f2007200640 1 Processor -- start 2024-07-26T06:16:57.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.788+0000 7f2007200640 1 -- start start 2024-07-26T06:16:57.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.788+0000 7f2007200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f2008149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.788+0000 7f2007200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f200814ce60 0x7f200814d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2007200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20080afdb0 0x7f20080b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2007200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20080b2130 con 0x7f1fe8004d40 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2007200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f200806f500 con 0x7f20080afdb0 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2007200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20080b22b0 con 0x7f200814ce60 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f2008149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f2008149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41426/0 (socket says 172.21.15.163:41426) 2024-07-26T06:16:57.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 -- 172.21.15.163:0/1511921339 learned_addr learned my addr 172.21.15.163:0/1511921339 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f1ffd400640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f200814ce60 0x7f200814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2006800640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20080afdb0 0x7f20080b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 -- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f200814ce60 msgr2=0x7f200814d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f200814ce60 0x7f200814d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 -- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20080afdb0 msgr2=0x7f20080b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20080afdb0 0x7f20080b0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.789+0000 7f2005e00640 1 -- 172.21.15.163:0/1511921339 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2008149f50 con 0x7f1fe8004d40 2024-07-26T06:16:57.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.790+0000 7f2005e00640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f2008149a10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800c6e0 tx=0x7f1ff800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:57.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.790+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8017070 con 0x7f1fe8004d40 2024-07-26T06:16:57.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.790+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ff8005360 con 0x7f1fe8004d40 2024-07-26T06:16:57.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.790+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8003850 con 0x7f1fe8004d40 2024-07-26T06:16:57.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.790+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20081f2ea0 con 0x7f1fe8004d40 2024-07-26T06:16:57.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.790+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20081f3460 con 0x7f1fe8004d40 2024-07-26T06:16:57.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.791+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20080b0770 con 0x7f1fe8004d40 2024-07-26T06:16:57.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.792+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f1ff801a070 con 0x7f1fe8004d40 2024-07-26T06:16:57.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.793+0000 7f1ffea00640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1fe4076480 0x7f1fe4078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:57.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.793+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7f1ff8096db0 con 0x7f1fe8004d40 2024-07-26T06:16:57.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.794+0000 7f1ffd400640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1fe4076480 0x7f1fe4078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:57.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.795+0000 7f1ffd400640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1fe4076480 0x7f1fe4078940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f200814f340 tx=0x7f1ff0005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:57.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:57.797+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ff809b070 con 0x7f1fe8004d40 2024-07-26T06:16:58.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.047+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20080b2c90 con 0x7f1fe8004d40 2024-07-26T06:16:58.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.048+0000 7f1ffea00640 1 -- 172.21.15.163:0/1511921339 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+3917 (secure 0 0 0) 0x7f1ff80606f0 con 0x7f1fe8004d40 2024-07-26T06:16:58.048 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:58.048 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":28,"btime":"2024-07-26T06:16:50:941243+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2030213828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2030213828},{"type":"v1","addr":"172.21.15.164:6835","nonce":2030213828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":4864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/2426427084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":2426427084},{"type":"v1","addr":"172.21.15.163:6835","nonce":2426427084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":4876,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/82388903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":82388903},{"type":"v1","addr":"172.21.15.164:6837","nonce":82388903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3190825707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3190825707},{"type":"v1","addr":"172.21.15.163:6834","nonce":3190825707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-07-26T06:16:58.048 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 28 2024-07-26T06:16:58.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1fe4076480 msgr2=0x7f1fe4078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:58.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1fe4076480 0x7f1fe4078940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f200814f340 tx=0x7f1ff0005f90 comp rx=0 tx=0).stop 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 msgr2=0x7f2008149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f2008149a10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800c6e0 tx=0x7f1ff800cba0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 shutdown_connections 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1fe4076480 0x7f1fe4078940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20080afdb0 0x7f20080b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f200814ce60 0x7f200814d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 --2- 172.21.15.163:0/1511921339 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fe8004d40 0x7f2008149a10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.051+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 >> 172.21.15.163:0/1511921339 conn(0x7f20080b0e30 msgr2=0x7f200815bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.052+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 shutdown_connections 2024-07-26T06:16:58.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.052+0000 7f2007200640 1 -- 172.21.15.163:0/1511921339 wait complete. 2024-07-26T06:16:58.072 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:16:58.072 INFO:tasks.ceph.mds.a:Stopping old one... 2024-07-26T06:16:58.072 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:16:58.072 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:16:58.073 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:16:58.073 INFO:tasks.ceph.mds.c:Stopping old one... 2024-07-26T06:16:58.073 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:16:58.073 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:16:58.073 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:16:58.073 INFO:tasks.ceph.mds.b:Stopping old one... 2024-07-26T06:16:58.074 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:16:58.074 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:16:58.074 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:16:58.074 INFO:tasks.ceph.mds.d:Stopping old one... 2024-07-26T06:16:58.074 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:16:58.074 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:16:58.093 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:16:58.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:16:58.094 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:16:58.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:16:58.097 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:16:58.098 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:16:58.099 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:16:58.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:16:58.140 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:16:58.140 DEBUG:teuthology.parallel:result is None 2024-07-26T06:16:58.145 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:16:58.146 DEBUG:teuthology.parallel:result is None 2024-07-26T06:16:58.146 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:16:58.146 DEBUG:teuthology.parallel:result is None 2024-07-26T06:16:58.152 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:16:58.152 DEBUG:teuthology.parallel:result is None 2024-07-26T06:16:58.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-07-26T06:16:58.290 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:16:58.291 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:16:58.301 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:16:58.304 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 -- 172.21.15.163:0/223558799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b806f350 msgr2=0x7fe1b80b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 --2- 172.21.15.163:0/223558799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b806f350 0x7fe1b80b2f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8069760 tx=0x7fe1a802b800 comp rx=0 tx=0).stop 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 -- 172.21.15.163:0/223558799 shutdown_connections 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 --2- 172.21.15.163:0/223558799 >> v2:172.21.15.163:3300/0 conn(0x7fe1b815cef0 0x7fe1b815d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 --2- 172.21.15.163:0/223558799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 0x7fe1b80b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 --2- 172.21.15.163:0/223558799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b806f350 0x7fe1b80b2f90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 -- 172.21.15.163:0/223558799 >> 172.21.15.163:0/223558799 conn(0x7fe1b806d420 msgr2=0x7fe1b806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:58.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 -- 172.21.15.163:0/223558799 shutdown_connections 2024-07-26T06:16:58.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.333+0000 7fe1be200640 1 -- 172.21.15.163:0/223558799 wait complete. 2024-07-26T06:16:58.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.335+0000 7fe1be200640 1 Processor -- start 2024-07-26T06:16:58.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.335+0000 7fe1be200640 1 -- start start 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1be200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 0x7fe1b8143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1be200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 0x7fe1b8157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1be200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b815cef0 0x7fe1b8157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1be200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b806bf40 con 0x7fe1b806f350 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1be200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b815dc40 con 0x7fe1b815cef0 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1be200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b806c0c0 con 0x7fe1b8159380 2024-07-26T06:16:58.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 0x7fe1b8143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 0x7fe1b8143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41448/0 (socket says 172.21.15.163:41448) 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 -- 172.21.15.163:0/1796840443 learned_addr learned my addr 172.21.15.163:0/1796840443 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7e00640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b815cef0 0x7fe1b8157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b6a00640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 0x7fe1b8157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 -- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 msgr2=0x7fe1b8157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 0x7fe1b8157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 -- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b815cef0 msgr2=0x7fe1b8157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b815cef0 0x7fe1b8157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b7400640 1 -- 172.21.15.163:0/1796840443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b81e6000 con 0x7fe1b806f350 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.336+0000 7fe1b6a00640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 0x7fe1b8157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.337+0000 7fe1b7400640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 0x7fe1b8143c50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800a960 tx=0x7fe1a80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:58.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.337+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8040070 con 0x7fe1b806f350 2024-07-26T06:16:58.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.337+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1a80047a0 con 0x7fe1b806f350 2024-07-26T06:16:58.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.337+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b81e62e0 con 0x7fe1b806f350 2024-07-26T06:16:58.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.337+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8037040 con 0x7fe1b806f350 2024-07-26T06:16:58.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.337+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b81e68a0 con 0x7fe1b806f350 2024-07-26T06:16:58.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.338+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b806c570 con 0x7fe1b806f350 2024-07-26T06:16:58.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.339+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fe1a8043070 con 0x7fe1b806f350 2024-07-26T06:16:58.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.341+0000 7fe1af400640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe1980764b0 0x7fe198078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:58.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.341+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4943+0+0 (secure 0 0 0) 0x7fe1a80b84b0 con 0x7fe1b806f350 2024-07-26T06:16:58.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.341+0000 7fe1b6a00640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe1980764b0 0x7fe198078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:58.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.342+0000 7fe1b6a00640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe1980764b0 0x7fe198078970 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe1b81e7340 tx=0x7fe1a4005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:58.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.345+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe1a8030070 con 0x7fe1b806f350 2024-07-26T06:16:58.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:58.565+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fe1b8074410 con 0x7fe1b806f350 2024-07-26T06:16:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.213+0000 7fe1af400640 1 -- 172.21.15.163:0/1796840443 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) ==== 97+0+0 (secure 0 0 0) 0x7fe1a8081ce0 con 0x7fe1b806f350 2024-07-26T06:16:59.213 INFO:teuthology.orchestra.run.smithi163.stderr: removed all blocklist entries 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe1980764b0 msgr2=0x7fe198078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe1980764b0 0x7fe198078970 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe1b81e7340 tx=0x7fe1a4005f90 comp rx=0 tx=0).stop 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 msgr2=0x7fe1b8143c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 0x7fe1b8143c50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800a960 tx=0x7fe1a80038a0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 shutdown_connections 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe1980764b0 0x7fe198078970 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b815cef0 0x7fe1b8157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b8159380 0x7fe1b8157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 --2- 172.21.15.163:0/1796840443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1b806f350 0x7fe1b8143c50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 >> 172.21.15.163:0/1796840443 conn(0x7fe1b806d420 msgr2=0x7fe1b80cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:59.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 shutdown_connections 2024-07-26T06:16:59.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.217+0000 7fe1be200640 1 -- 172.21.15.163:0/1796840443 wait complete. 2024-07-26T06:16:59.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.441+0000 7f43eee00640 1 -- 172.21.15.163:0/376174536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e815b230 msgr2=0x7f43e806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.441+0000 7f43eee00640 1 --2- 172.21.15.163:0/376174536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e815b230 0x7f43e806ee30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005fd0 tx=0x7f43d802a530 comp rx=0 tx=0).stop 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.442+0000 7f43eee00640 1 -- 172.21.15.163:0/376174536 shutdown_connections 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.442+0000 7f43eee00640 1 --2- 172.21.15.163:0/376174536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43e806bf40 0x7f43e806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.442+0000 7f43eee00640 1 --2- 172.21.15.163:0/376174536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e815b230 0x7f43e806ee30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.442+0000 7f43eee00640 1 -- 172.21.15.163:0/376174536 >> 172.21.15.163:0/376174536 conn(0x7f43e80b1f50 msgr2=0x7f43e80b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.442+0000 7f43eee00640 1 -- 172.21.15.163:0/376174536 shutdown_connections 2024-07-26T06:16:59.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.442+0000 7f43eee00640 1 -- 172.21.15.163:0/376174536 wait complete. 2024-07-26T06:16:59.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.444+0000 7f43eee00640 1 Processor -- start 2024-07-26T06:16:59.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.444+0000 7f43eee00640 1 -- start start 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43eee00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 0x7f43e81569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43eee00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e8072380 0x7f43e81e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43eee00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43e815b230 0x7f43e81e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43eee00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43e815b790 con 0x7f43e806bf40 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43eee00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43e815b610 con 0x7f43e8072380 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43eee00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f43e8072f20 con 0x7f43e815b230 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 0x7f43e81569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 0x7f43e81569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41472/0 (socket says 172.21.15.163:41472) 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 -- 172.21.15.163:0/3646041795 learned_addr learned my addr 172.21.15.163:0/3646041795 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7400640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e8072380 0x7f43e81e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43ece00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43e815b230 0x7f43e81e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 -- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43e815b230 msgr2=0x7f43e81e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43e815b230 0x7f43e81e6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 -- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e8072380 msgr2=0x7f43e81e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.445+0000 7f43e7e00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e8072380 0x7f43e81e66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43e7e00640 1 -- 172.21.15.163:0/3646041795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e81e5ad0 con 0x7f43e806bf40 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43e7e00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 0x7f43e81569d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f43d8000c00 tx=0x7f43d80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d803f070 con 0x7f43e806bf40 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43d8004890 con 0x7f43e806bf40 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8035030 con 0x7f43e806bf40 2024-07-26T06:16:59.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e81e4a30 con 0x7f43e806bf40 2024-07-26T06:16:59.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.446+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e81e4ff0 con 0x7f43e806bf40 2024-07-26T06:16:59.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.448+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e806df10 con 0x7f43e806bf40 2024-07-26T06:16:59.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.449+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f43d8042070 con 0x7f43e806bf40 2024-07-26T06:16:59.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.450+0000 7f43dfe00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43c8076480 0x7f43c8078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.450+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7f43d80b86d0 con 0x7f43e806bf40 2024-07-26T06:16:59.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.450+0000 7f43e7400640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43c8076480 0x7f43c8078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.452+0000 7f43e7400640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43c8076480 0x7f43c8078940 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f43e81e3c90 tx=0x7f43d4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:59.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.454+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43e806df10 con 0x7f43e806bf40 2024-07-26T06:16:59.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.741+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f43e80b4000 con 0x7f43e806bf40 2024-07-26T06:16:59.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.742+0000 7f43dfe00640 1 -- 172.21.15.163:0/3646041795 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7f43d8081ff0 con 0x7f43e806bf40 2024-07-26T06:16:59.742 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:16:59.742 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_dump": [ 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.a", 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA==", 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.743 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.b", 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg==", 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.744 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.746 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.c", 2024-07-26T06:16:59.746 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw==", 2024-07-26T06:16:59.747 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.747 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.747 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:16:59.747 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:16:59.747 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.748 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.d", 2024-07-26T06:16:59.748 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg==", 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.749 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.0", 2024-07-26T06:16:59.750 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA==", 2024-07-26T06:16:59.750 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.750 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.750 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.750 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.1", 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA==", 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.2", 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA==", 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.3", 2024-07-26T06:16:59.752 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg==", 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.4", 2024-07-26T06:16:59.753 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA==", 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.5", 2024-07-26T06:16:59.754 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g==", 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.755 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.6", 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ==", 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.7", 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA==", 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.0", 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew==", 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.757 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.1", 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ==", 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.758 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.2", 2024-07-26T06:16:59.759 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw==", 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.3", 2024-07-26T06:16:59.760 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw==", 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.admin", 2024-07-26T06:16:59.761 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm+ibKBxAA1ZBGvVy5rNIzMWwOFwFJFw==", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.x", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ==", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:16:59.762 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.y", 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ==", 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:16:59.763 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:16:59.764 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.764 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:16:59.764 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:16:59.764 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:16:59.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.745+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43c8076480 msgr2=0x7f43c8078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43c8076480 0x7f43c8078940 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f43e81e3c90 tx=0x7f43d4008040 comp rx=0 tx=0).stop 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 msgr2=0x7f43e81569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 0x7f43e81569d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f43d8000c00 tx=0x7f43d80039e0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 shutdown_connections 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f43c8076480 0x7f43c8078940 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43e815b230 0x7f43e81e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e8072380 0x7f43e81e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 --2- 172.21.15.163:0/3646041795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e806bf40 0x7f43e81569d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 >> 172.21.15.163:0/3646041795 conn(0x7f43e80b1f50 msgr2=0x7f43e80b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 shutdown_connections 2024-07-26T06:16:59.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.746+0000 7f43eee00640 1 -- 172.21.15.163:0/3646041795 wait complete. 2024-07-26T06:16:59.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:16:59.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a9400640 1 -- 172.21.15.163:0/3564398546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4288003a00 msgr2=0x7f4288003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a9400640 1 --2- 172.21.15.163:0/3564398546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4288003a00 0x7f4288003e20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4294009b50 tx=0x7f4288005830 comp rx=0 tx=0).stop 2024-07-26T06:16:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a3200640 1 -- 172.21.15.163:0/3564398546 reap_dead start 2024-07-26T06:16:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a9400640 1 -- 172.21.15.163:0/3564398546 shutdown_connections 2024-07-26T06:16:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a9400640 1 -- 172.21.15.163:0/3564398546 >> 172.21.15.163:0/3564398546 conn(0x7f42a406d420 msgr2=0x7f42a406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:16:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a9400640 1 -- 172.21.15.163:0/3564398546 shutdown_connections 2024-07-26T06:16:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.975+0000 7f42a9400640 1 -- 172.21.15.163:0/3564398546 wait complete. 2024-07-26T06:16:59.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.977+0000 7f42a9400640 1 Processor -- start 2024-07-26T06:16:59.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.977+0000 7f42a9400640 1 -- start start 2024-07-26T06:16:59.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a9400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4288003a00 0x7f42a41f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a9400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42a41e6480 0x7f42a41f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 0x7f42a40b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a9400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f42a4143590 con 0x7f42a40b1010 2024-07-26T06:16:59.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a9400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42a40cef30 con 0x7f4288003a00 2024-07-26T06:16:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a9400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42a4143710 con 0x7f42a41e6480 2024-07-26T06:16:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.978+0000 7f42a3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 0x7f42a40b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 0x7f42a40b1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41490/0 (socket says 172.21.15.163:41490) 2024-07-26T06:16:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 -- 172.21.15.163:0/1133727835 learned_addr learned my addr 172.21.15.163:0/1133727835 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:16:59.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f429be00640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42a41e6480 0x7f42a41f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a2800640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4288003a00 0x7f42a41f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 -- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42a41e6480 msgr2=0x7f42a41f1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42a41e6480 0x7f42a41f1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 -- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4288003a00 msgr2=0x7f42a41f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:16:59.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4288003a00 0x7f42a41f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 -- 172.21.15.163:0/1133727835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a41e6900 con 0x7f42a40b1010 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f42a3200640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 0x7f42a40b1490 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f42a4142db0 tx=0x7f429c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c017070 con 0x7f42a40b1010 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f429c00ad60 con 0x7f42a40b1010 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.979+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c0072d0 con 0x7f42a40b1010 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.980+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a41e6bb0 con 0x7f42a40b1010 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.980+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a41e7170 con 0x7f42a40b1010 2024-07-26T06:16:59.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.981+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a40b19d0 con 0x7f42a40b1010 2024-07-26T06:16:59.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.982+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f429c01a070 con 0x7f42a40b1010 2024-07-26T06:16:59.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.983+0000 7f429aa00640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4280076480 0x7f4280078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:16:59.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.984+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7f429c097460 con 0x7f42a40b1010 2024-07-26T06:16:59.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.984+0000 7f42a2800640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4280076480 0x7f4280078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:16:59.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.985+0000 7f42a2800640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4280076480 0x7f4280078940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4294035000 tx=0x7f429402d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:16:59.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:16:59.988+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f429c060e90 con 0x7f42a40b1010 2024-07-26T06:17:00.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.193+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f42a406c360 con 0x7f42a40b1010 2024-07-26T06:17:00.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.194+0000 7f429aa00640 1 -- 172.21.15.163:0/1133727835 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) ==== 84+0+65 (secure 0 0 0) 0x7f429c060790 con 0x7f42a40b1010 2024-07-26T06:17:00.195 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:00.195 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:17:00.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:00.196 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:17:00.196 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:17:00.196 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:00.196 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:17:00.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4280076480 msgr2=0x7f4280078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:00.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4280076480 0x7f4280078940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4294035000 tx=0x7f429402d040 comp rx=0 tx=0).stop 2024-07-26T06:17:00.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 msgr2=0x7f42a40b1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:00.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 0x7f42a40b1490 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f42a4142db0 tx=0x7f429c00a060 comp rx=0 tx=0).stop 2024-07-26T06:17:00.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 shutdown_connections 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4280076480 0x7f4280078940 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f42a40b1010 0x7f42a40b1490 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42a41e6480 0x7f42a41f1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 --2- 172.21.15.163:0/1133727835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4288003a00 0x7f42a41f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 >> 172.21.15.163:0/1133727835 conn(0x7f42a406d420 msgr2=0x7f42a40b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.197+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 shutdown_connections 2024-07-26T06:17:00.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.198+0000 7f42a9400640 1 -- 172.21.15.163:0/1133727835 wait complete. 2024-07-26T06:17:00.220 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:17:00.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:17:00.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 -- 172.21.15.163:0/2477835656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a815d5a0 msgr2=0x7f57a80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:00.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 --2- 172.21.15.163:0/2477835656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a815d5a0 0x7f57a80ce2f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5798009830 tx=0x7f579802a480 comp rx=0 tx=0).stop 2024-07-26T06:17:00.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 -- 172.21.15.163:0/2477835656 shutdown_connections 2024-07-26T06:17:00.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 --2- 172.21.15.163:0/2477835656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a815d5a0 0x7f57a80ce2f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 -- 172.21.15.163:0/2477835656 >> 172.21.15.163:0/2477835656 conn(0x7f57a80cfda0 msgr2=0x7f57a80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 -- 172.21.15.163:0/2477835656 shutdown_connections 2024-07-26T06:17:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.424+0000 7f57ad000640 1 -- 172.21.15.163:0/2477835656 wait complete. 2024-07-26T06:17:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.426+0000 7f57ad000640 1 Processor -- start 2024-07-26T06:17:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.426+0000 7f57ad000640 1 -- start start 2024-07-26T06:17:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.427+0000 7f57ad000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 0x7f57a80ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.427+0000 7f57ad000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57a80ce830 0x7f57a814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.427+0000 7f57ad000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57a815d5a0 0x7f57a814d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.427+0000 7f57ad000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57a80b6f60 con 0x7f57a80b1e80 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.427+0000 7f57ad000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f57a80b6de0 con 0x7f57a815d5a0 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.427+0000 7f57ad000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f57a80b54b0 con 0x7f57a80ce830 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 0x7f57a80ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 0x7f57a80ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41510/0 (socket says 172.21.15.163:41510) 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 -- 172.21.15.163:0/4003213785 learned_addr learned my addr 172.21.15.163:0/4003213785 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7e00640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57a815d5a0 0x7f57a814d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 -- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57a80ce830 msgr2=0x7f57a814d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57a80ce830 0x7f57a814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 -- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57a815d5a0 msgr2=0x7f57a814d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57a815d5a0 0x7f57a814d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 -- 172.21.15.163:0/4003213785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a80cb0c0 con 0x7f57a80b1e80 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7e00640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57a815d5a0 0x7f57a814d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.428+0000 7f57a7400640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 0x7f57a80ccd50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5798000c00 tx=0x7f5798033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.429+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798031d60 con 0x7f57a80b1e80 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.429+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5798033070 con 0x7f57a80b1e80 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.429+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57980334c0 con 0x7f57a80b1e80 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.429+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a80cb3a0 con 0x7f57a80b1e80 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.429+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a80cbc40 con 0x7f57a80b1e80 2024-07-26T06:17:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.430+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a80cb790 con 0x7f57a80b1e80 2024-07-26T06:17:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.431+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f5798033660 con 0x7f57a80b1e80 2024-07-26T06:17:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.432+0000 7f579f400640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5788076480 0x7f5788078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.433+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7f57980b8d00 con 0x7f57a80b1e80 2024-07-26T06:17:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.433+0000 7f57a6a00640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5788076480 0x7f5788078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:00.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.434+0000 7f57a6a00640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5788076480 0x7f5788078940 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f57a80d27f0 tx=0x7f57940079b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:00.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.437+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f579803f030 con 0x7f57a80b1e80 2024-07-26T06:17:00.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:00.644+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f57a80c6b40 con 0x7f57a80b1e80 2024-07-26T06:17:01.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.235+0000 7f579f400640 1 -- 172.21.15.163:0/4003213785 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) ==== 138+0+0 (secure 0 0 0) 0x7f5798082580 con 0x7f57a80b1e80 2024-07-26T06:17:01.237 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:17:01.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.239+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5788076480 msgr2=0x7f5788078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:01.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.239+0000 7f57ad000640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5788076480 0x7f5788078940 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f57a80d27f0 tx=0x7f57940079b0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.239+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 msgr2=0x7f57a80ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.239+0000 7f57ad000640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 0x7f57a80ccd50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5798000c00 tx=0x7f5798033d60 comp rx=0 tx=0).stop 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 shutdown_connections 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5788076480 0x7f5788078940 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57a815d5a0 0x7f57a814d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57a80ce830 0x7f57a814d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 --2- 172.21.15.163:0/4003213785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57a80b1e80 0x7f57a80ccd50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 >> 172.21.15.163:0/4003213785 conn(0x7f57a80cfda0 msgr2=0x7f57a8152860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:01.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 shutdown_connections 2024-07-26T06:17:01.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.240+0000 7f57ad000640 1 -- 172.21.15.163:0/4003213785 wait complete. 2024-07-26T06:17:01.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 -- 172.21.15.163:0/2364731901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea780b1e80 msgr2=0x7fea780b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 --2- 172.21.15.163:0/2364731901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea780b1e80 0x7fea780b6bd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fea70005fd0 tx=0x7fea7002a8e0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 -- 172.21.15.163:0/2364731901 shutdown_connections 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 --2- 172.21.15.163:0/2364731901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea780b1e80 0x7fea780b6bd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 --2- 172.21.15.163:0/2364731901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea7815d5a0 0x7fea780ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 -- 172.21.15.163:0/2364731901 >> 172.21.15.163:0/2364731901 conn(0x7fea780cfda0 msgr2=0x7fea780d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:01.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 -- 172.21.15.163:0/2364731901 shutdown_connections 2024-07-26T06:17:01.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.465+0000 7fea7e600640 1 -- 172.21.15.163:0/2364731901 wait complete. 2024-07-26T06:17:01.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.467+0000 7fea7e600640 1 Processor -- start 2024-07-26T06:17:01.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.467+0000 7fea7e600640 1 -- start start 2024-07-26T06:17:01.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7e600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea780b1e80 0x7fea781f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:01.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7e600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea780ce830 0x7fea781f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:01.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7e600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 0x7fea780c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7e600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fea780b5130 con 0x7fea7815d5a0 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7e600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fea780b4fb0 con 0x7fea780ce830 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7e600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fea780b52b0 con 0x7fea780b1e80 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7dc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 0x7fea780c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7dc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 0x7fea780c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41530/0 (socket says 172.21.15.163:41530) 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7dc00640 1 -- 172.21.15.163:0/3111205102 learned_addr learned my addr 172.21.15.163:0/3111205102 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea7d200640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea780b1e80 0x7fea781f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.468+0000 7fea77e00640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea780ce830 0x7fea781f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7dc00640 1 -- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea780b1e80 msgr2=0x7fea781f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7dc00640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea780b1e80 0x7fea781f1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7dc00640 1 -- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea780ce830 msgr2=0x7fea781f46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:01.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7dc00640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea780ce830 0x7fea781f46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7dc00640 1 -- 172.21.15.163:0/3111205102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7815f200 con 0x7fea7815d5a0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7dc00640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 0x7fea780c9200 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fea70009d90 tx=0x7fea70003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70004240 con 0x7fea7815d5a0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea700043e0 con 0x7fea7815d5a0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70004830 con 0x7fea7815d5a0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.469+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7815f4e0 con 0x7fea7815d5a0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.470+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7815faa0 con 0x7fea7815d5a0 2024-07-26T06:17:01.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.471+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea780ce2f0 con 0x7fea7815d5a0 2024-07-26T06:17:01.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.472+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fea70031470 con 0x7fea7815d5a0 2024-07-26T06:17:01.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.473+0000 7fea74c00640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea580764b0 0x7fea58078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:01.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.474+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5227+0+0 (secure 0 0 0) 0x7fea700b94f0 con 0x7fea7815d5a0 2024-07-26T06:17:01.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.474+0000 7fea7d200640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea580764b0 0x7fea58078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:01.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.475+0000 7fea7d200640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea580764b0 0x7fea58078970 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0058c0 tx=0x7fea6c005d80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:01.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.477+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fea70082d90 con 0x7fea7815d5a0 2024-07-26T06:17:01.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:01.686+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fea7815b420 con 0x7fea7815d5a0 2024-07-26T06:17:02.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.242+0000 7fea74c00640 1 -- 172.21.15.163:0/3111205102 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) ==== 170+0+0 (secure 0 0 0) 0x7fea70082690 con 0x7fea7815d5a0 2024-07-26T06:17:02.243 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.245+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea580764b0 msgr2=0x7fea58078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.245+0000 7fea7e600640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea580764b0 0x7fea58078970 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0058c0 tx=0x7fea6c005d80 comp rx=0 tx=0).stop 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 msgr2=0x7fea780c9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 0x7fea780c9200 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fea70009d90 tx=0x7fea70003f10 comp rx=0 tx=0).stop 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 shutdown_connections 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea580764b0 0x7fea58078970 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fea7815d5a0 0x7fea780c9200 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea780ce830 0x7fea781f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 --2- 172.21.15.163:0/3111205102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea780b1e80 0x7fea781f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 >> 172.21.15.163:0/3111205102 conn(0x7fea780cfda0 msgr2=0x7fea7815c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:02.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 shutdown_connections 2024-07-26T06:17:02.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.246+0000 7fea7e600640 1 -- 172.21.15.163:0/3111205102 wait complete. 2024-07-26T06:17:02.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:17:02.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.462+0000 7feab0a00640 1 -- 172.21.15.163:0/1179960388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0b5250 msgr2=0x7feaac0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:02.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.462+0000 7feab0a00640 1 --2- 172.21.15.163:0/1179960388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0b5250 0x7feaac0b3a70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7feaac073680 tx=0x7fea9402a4d0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 -- 172.21.15.163:0/1179960388 shutdown_connections 2024-07-26T06:17:02.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 --2- 172.21.15.163:0/1179960388 >> v2:172.21.15.164:3300/0 conn(0x7feaac0b5660 0x7feaac0b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 --2- 172.21.15.163:0/1179960388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0b5250 0x7feaac0b3a70 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 --2- 172.21.15.163:0/1179960388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaac15b3c0 0x7feaac0b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 -- 172.21.15.163:0/1179960388 >> 172.21.15.163:0/1179960388 conn(0x7feaac0b6a60 msgr2=0x7feaac0b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:02.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 -- 172.21.15.163:0/1179960388 shutdown_connections 2024-07-26T06:17:02.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.463+0000 7feab0a00640 1 -- 172.21.15.163:0/1179960388 wait complete. 2024-07-26T06:17:02.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.464+0000 7feab0a00640 1 Processor -- start 2024-07-26T06:17:02.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.465+0000 7feab0a00640 1 -- start start 2024-07-26T06:17:02.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feab0a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaac0b5660 0x7feaac1f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:02.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feab0a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaac1ee010 0x7feaac0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:02.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feab0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 0x7feaac0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:02.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feab0a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7feaac0d16b0 con 0x7feaac0cfd30 2024-07-26T06:17:02.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feab0a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feaac0d1530 con 0x7feaac1ee010 2024-07-26T06:17:02.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feab0a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feaac0d1830 con 0x7feaac0b5660 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 0x7feaac0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 0x7feaac0d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41536/0 (socket says 172.21.15.163:41536) 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 -- 172.21.15.163:0/1328185323 learned_addr learned my addr 172.21.15.163:0/1328185323 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaa9e00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaac0b5660 0x7feaac1f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaa3e00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaac1ee010 0x7feaac0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 -- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaac0b5660 msgr2=0x7feaac1f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaac0b5660 0x7feaac1f4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 -- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaac1ee010 msgr2=0x7feaac0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:02.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaac1ee010 0x7feaac0ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:02.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.466+0000 7feaaa800640 1 -- 172.21.15.163:0/1328185323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac0ade20 con 0x7feaac0cfd30 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.467+0000 7feaaa800640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 0x7feaac0d01e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7feaa400cce0 tx=0x7feaa400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.467+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa400f450 con 0x7feaac0cfd30 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.467+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaa400f700 con 0x7feaac0cfd30 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.467+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac0ae0f0 con 0x7feaac0cfd30 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.467+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa400fb50 con 0x7feaac0cfd30 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.467+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac0ae6b0 con 0x7feaac0cfd30 2024-07-26T06:17:02.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.468+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaac0b5a90 con 0x7feaac0cfd30 2024-07-26T06:17:02.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.470+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7feaa400fcf0 con 0x7feaac0cfd30 2024-07-26T06:17:02.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.471+0000 7feaa2000640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea880764b0 0x7fea88078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:02.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.471+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5609+0+0 (secure 0 0 0) 0x7feaa40985d0 con 0x7feaac0cfd30 2024-07-26T06:17:02.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.471+0000 7feaa9e00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea880764b0 0x7fea88078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:02.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.472+0000 7feaa9e00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea880764b0 0x7fea88078970 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feaac0d1d40 tx=0x7fea9c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:02.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.476+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feaa4014030 con 0x7feaac0cfd30 2024-07-26T06:17:02.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:02.715+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7feaac1593b0 con 0x7feaac0cfd30 2024-07-26T06:17:03.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.269+0000 7feaa2000640 1 -- 172.21.15.163:0/1328185323 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v31) ==== 172+0+0 (secure 0 0 0) 0x7feaa4061cc0 con 0x7feaac0cfd30 2024-07-26T06:17:03.271 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 8 and data pool 9 2024-07-26T06:17:03.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea880764b0 msgr2=0x7fea88078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea880764b0 0x7fea88078970 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feaac0d1d40 tx=0x7fea9c008040 comp rx=0 tx=0).stop 2024-07-26T06:17:03.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 msgr2=0x7feaac0d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 0x7feaac0d01e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7feaa400cce0 tx=0x7feaa400a620 comp rx=0 tx=0).stop 2024-07-26T06:17:03.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 shutdown_connections 2024-07-26T06:17:03.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fea880764b0 0x7fea88078970 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feaac0cfd30 0x7feaac0d01e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaac1ee010 0x7feaac0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 --2- 172.21.15.163:0/1328185323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaac0b5660 0x7feaac1f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 >> 172.21.15.163:0/1328185323 conn(0x7feaac0b6a60 msgr2=0x7feaac0cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 shutdown_connections 2024-07-26T06:17:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.273+0000 7feab0a00640 1 -- 172.21.15.163:0/1328185323 wait complete. 2024-07-26T06:17:03.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a2aa00640 1 -- 172.21.15.163:0/1686061419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a08003aa0 msgr2=0x7f3a08003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/1686061419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a08003aa0 0x7f3a08003ec0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400baf0 tx=0x7f3a08005830 comp rx=0 tx=0).stop 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a28a00640 1 -- 172.21.15.163:0/1686061419 reap_dead start 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a2aa00640 1 -- 172.21.15.163:0/1686061419 shutdown_connections 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a2aa00640 1 -- 172.21.15.163:0/1686061419 >> 172.21.15.163:0/1686061419 conn(0x7f3a2406d420 msgr2=0x7f3a2406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a2aa00640 1 -- 172.21.15.163:0/1686061419 shutdown_connections 2024-07-26T06:17:03.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.485+0000 7f3a2aa00640 1 -- 172.21.15.163:0/1686061419 wait complete. 2024-07-26T06:17:03.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.486+0000 7f3a2aa00640 1 Processor -- start 2024-07-26T06:17:03.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.486+0000 7f3a2aa00640 1 -- start start 2024-07-26T06:17:03.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.487+0000 7f3a2aa00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a08003aa0 0x7f3a24150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.487+0000 7f3a2aa00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a241f1730 0x7f3a24151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.487+0000 7f3a2aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 0x7f3a24151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.487+0000 7f3a2aa00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2415be40 con 0x7f3a2414bc80 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.487+0000 7f3a2aa00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2415bfc0 con 0x7f3a08003aa0 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.487+0000 7f3a2aa00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2415c140 con 0x7f3a241f1730 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 0x7f3a24151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 0x7f3a24151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41540/0 (socket says 172.21.15.163:41540) 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 -- 172.21.15.163:0/2236807322 learned_addr learned my addr 172.21.15.163:0/2236807322 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a1b400640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a241f1730 0x7f3a24151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a23e00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a08003aa0 0x7f3a24150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 -- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a241f1730 msgr2=0x7f3a24151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a241f1730 0x7f3a24151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 -- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a08003aa0 msgr2=0x7f3a24150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a08003aa0 0x7f3a24150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 -- 172.21.15.163:0/2236807322 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a24152b40 con 0x7f3a2414bc80 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.488+0000 7f3a28a00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 0x7f3a24151d40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c001330 tx=0x7f3a1c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.489+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00b840 con 0x7f3a2414bc80 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.489+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a1c017070 con 0x7f3a2414bc80 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.489+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00bca0 con 0x7f3a2414bc80 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.489+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a24150b80 con 0x7f3a2414bc80 2024-07-26T06:17:03.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.489+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2414d570 con 0x7f3a2414bc80 2024-07-26T06:17:03.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.490+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2415ed20 con 0x7f3a2414bc80 2024-07-26T06:17:03.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.491+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f3a1c01c460 con 0x7f3a2414bc80 2024-07-26T06:17:03.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.492+0000 7f3a20c00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3a00076480 0x7f3a00078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.492+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f3a1c097c50 con 0x7f3a2414bc80 2024-07-26T06:17:03.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.492+0000 7f3a23e00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3a00076480 0x7f3a00078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.494+0000 7f3a23e00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3a00076480 0x7f3a00078940 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a14036000 tx=0x7f3a140046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:03.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.496+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a1c061320 con 0x7f3a2414bc80 2024-07-26T06:17:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.700+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a24069050 con 0x7f3a2414bc80 2024-07-26T06:17:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.702+0000 7f3a20c00640 1 -- 172.21.15.163:0/2236807322 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16702 (secure 0 0 0) 0x7f3a1c069310 con 0x7f3a2414bc80 2024-07-26T06:17:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:03.705 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":57,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:02.717593+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:00.658125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:01.704071+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:03.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3a00076480 msgr2=0x7f3a00078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3a00076480 0x7f3a00078940 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a14036000 tx=0x7f3a140046c0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 msgr2=0x7f3a24151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 0x7f3a24151d40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c001330 tx=0x7f3a1c00dc60 comp rx=0 tx=0).stop 2024-07-26T06:17:03.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 shutdown_connections 2024-07-26T06:17:03.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3a00076480 0x7f3a00078940 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a2414bc80 0x7f3a24151d40 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a241f1730 0x7f3a24151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 --2- 172.21.15.163:0/2236807322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a08003aa0 0x7f3a24150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 >> 172.21.15.163:0/2236807322 conn(0x7f3a2406d420 msgr2=0x7f3a240af720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:03.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 shutdown_connections 2024-07-26T06:17:03.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.705+0000 7f3a2aa00640 1 -- 172.21.15.163:0/2236807322 wait complete. 2024-07-26T06:17:03.726 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:03.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 -- 172.21.15.163:0/1499984006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9406bf40 msgr2=0x7f0f9406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1499984006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9406bf40 0x7f0f9406c370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c005fd0 tx=0x7f0f8c02aa00 comp rx=0 tx=0).stop 2024-07-26T06:17:03.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 -- 172.21.15.163:0/1499984006 shutdown_connections 2024-07-26T06:17:03.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1499984006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9406bf40 0x7f0f9406c370 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1499984006 >> v2:172.21.15.164:3300/0 conn(0x7f0f94071cf0 0x7f0f9415ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 -- 172.21.15.163:0/1499984006 >> 172.21.15.163:0/1499984006 conn(0x7f0f9406d420 msgr2=0x7f0f9406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:03.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 -- 172.21.15.163:0/1499984006 shutdown_connections 2024-07-26T06:17:03.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.921+0000 7f0f99a00640 1 -- 172.21.15.163:0/1499984006 wait complete. 2024-07-26T06:17:03.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.923+0000 7f0f99a00640 1 Processor -- start 2024-07-26T06:17:03.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.923+0000 7f0f99a00640 1 -- start start 2024-07-26T06:17:03.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f99a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9406bf40 0x7f0f94143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f99a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f94071cf0 0x7f0f941514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f99a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 0x7f0f940ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f99a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0f940736b0 con 0x7f0f940cb7e0 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f99a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94073530 con 0x7f0f9406bf40 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f99a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0f94073830 con 0x7f0f94071cf0 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f93400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 0x7f0f940ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f93400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 0x7f0f940ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41552/0 (socket says 172.21.15.163:41552) 2024-07-26T06:17:03.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f93400640 1 -- 172.21.15.163:0/1256220879 learned_addr learned my addr 172.21.15.163:0/1256220879 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:03.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f8be00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f94071cf0 0x7f0f941514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.924+0000 7f0f92a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9406bf40 0x7f0f94143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f93400640 1 -- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f94071cf0 msgr2=0x7f0f941514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f93400640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f94071cf0 0x7f0f941514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f93400640 1 -- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9406bf40 msgr2=0x7f0f94143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f93400640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9406bf40 0x7f0f94143ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f93400640 1 -- 172.21.15.163:0/1256220879 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9414fb60 con 0x7f0f940cb7e0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f93400640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 0x7f0f940ccd70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009d90 tx=0x7f0f8c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c047070 con 0x7f0f940cb7e0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f8c02f660 con 0x7f0f940cb7e0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f94151bd0 con 0x7f0f940cb7e0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.925+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c033050 con 0x7f0f940cb7e0 2024-07-26T06:17:03.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.926+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f94152190 con 0x7f0f940cb7e0 2024-07-26T06:17:03.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.927+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f94073d00 con 0x7f0f940cb7e0 2024-07-26T06:17:03.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.928+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f0f8c04a070 con 0x7f0f940cb7e0 2024-07-26T06:17:03.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.929+0000 7f0f8aa00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0f700764b0 0x7f0f70078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:03.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.929+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0f8c0b8bd0 con 0x7f0f940cb7e0 2024-07-26T06:17:03.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.929+0000 7f0f92a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0f700764b0 0x7f0f70078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:03.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.930+0000 7f0f92a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0f700764b0 0x7f0f70078970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0f94155dc0 tx=0x7f0f84009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:03.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:03.933+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f8c0821a0 con 0x7f0f940cb7e0 2024-07-26T06:17:04.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.126+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f940adcf0 con 0x7f0f940cb7e0 2024-07-26T06:17:04.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.129+0000 7f0f8aa00640 1 -- 172.21.15.163:0/1256220879 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16702 (secure 0 0 0) 0x7f0f8c081aa0 con 0x7f0f940cb7e0 2024-07-26T06:17:04.130 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:04.131 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":57,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:02.717593+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:00.658125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:01.704071+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:04.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0f700764b0 msgr2=0x7f0f70078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:04.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0f700764b0 0x7f0f70078970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0f94155dc0 tx=0x7f0f84009040 comp rx=0 tx=0).stop 2024-07-26T06:17:04.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 msgr2=0x7f0f940ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:04.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 0x7f0f940ccd70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009d90 tx=0x7f0f8c003f10 comp rx=0 tx=0).stop 2024-07-26T06:17:04.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 shutdown_connections 2024-07-26T06:17:04.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0f700764b0 0x7f0f70078970 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f940cb7e0 0x7f0f940ccd70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f94071cf0 0x7f0f941514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 --2- 172.21.15.163:0/1256220879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f9406bf40 0x7f0f94143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.131+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 >> 172.21.15.163:0/1256220879 conn(0x7f0f9406d420 msgr2=0x7f0f940b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:04.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.132+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 shutdown_connections 2024-07-26T06:17:04.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.132+0000 7f0f99a00640 1 -- 172.21.15.163:0/1256220879 wait complete. 2024-07-26T06:17:04.150 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:17:04.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 -- 172.21.15.163:0/2253947078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede00cdd90 msgr2=0x7fede00ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:04.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 --2- 172.21.15.163:0/2253947078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede00cdd90 0x7fede00ce170 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005fd0 tx=0x7fedd002a5d0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 -- 172.21.15.163:0/2253947078 shutdown_connections 2024-07-26T06:17:04.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 --2- 172.21.15.163:0/2253947078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede00cdd90 0x7fede00ce170 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 -- 172.21.15.163:0/2253947078 >> 172.21.15.163:0/2253947078 conn(0x7fede006bf40 msgr2=0x7fede006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:04.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 -- 172.21.15.163:0/2253947078 shutdown_connections 2024-07-26T06:17:04.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.345+0000 7fede7000640 1 -- 172.21.15.163:0/2253947078 wait complete. 2024-07-26T06:17:04.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.346+0000 7fede7000640 1 Processor -- start 2024-07-26T06:17:04.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.347+0000 7fede7000640 1 -- start start 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7fede7000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 0x7fede015a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7fede7000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede00cdd90 0x7fede015ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7fede7000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede01413b0 0x7fede01f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7fede7000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fede013ef70 con 0x7fede0074020 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7fede7000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fede013edf0 con 0x7fede00cdd90 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7fede7000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fede0074430 con 0x7fede01413b0 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 0x7fede015a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 0x7fede015a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41574/0 (socket says 172.21.15.163:41574) 2024-07-26T06:17:04.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 -- 172.21.15.163:0/3351255132 learned_addr learned my addr 172.21.15.163:0/3351255132 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:04.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 -- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede01413b0 msgr2=0x7fede01f78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:04.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede01413b0 0x7fede01f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 -- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede00cdd90 msgr2=0x7fede015ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:04.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede00cdd90 0x7fede015ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:04.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.348+0000 7feddfe00640 1 -- 172.21.15.163:0/3351255132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede00cd9b0 con 0x7fede0074020 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.349+0000 7feddfe00640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 0x7fede015a850 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006010 tx=0x7fedd00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.349+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd003f070 con 0x7fede0074020 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.349+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedd00047a0 con 0x7fede0074020 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.349+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede00cbd60 con 0x7fede0074020 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.349+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0035030 con 0x7fede0074020 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.349+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede00cd5d0 con 0x7fede0074020 2024-07-26T06:17:04.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.350+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede0069810 con 0x7fede0074020 2024-07-26T06:17:04.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.351+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fedd0042070 con 0x7fede0074020 2024-07-26T06:17:04.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.352+0000 7feddcc00640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fedbc076480 0x7fedbc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:04.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.352+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fedd00b88b0 con 0x7fede0074020 2024-07-26T06:17:04.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.353+0000 7fedd7400640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fedbc076480 0x7fedbc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:04.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.354+0000 7fedd7400640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fedbc076480 0x7fedbc078940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fede00cb370 tx=0x7fedc8006ba0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:04.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.356+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fedd0081de0 con 0x7fede0074020 2024-07-26T06:17:04.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:04.585+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fede00ce170 con 0x7fede0074020 2024-07-26T06:17:05.224 INFO:tasks.ceph.mds.d.smithi164.stderr:2024-07-26T06:17:05.223+0000 7fa2e6e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:17:05.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.318+0000 7feddcc00640 1 -- 172.21.15.163:0/3351255132 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7fedd00816e0 con 0x7fede0074020 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fedbc076480 msgr2=0x7fedbc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fedbc076480 0x7fedbc078940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fede00cb370 tx=0x7fedc8006ba0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 msgr2=0x7fede015a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 0x7fede015a850 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fedd0006010 tx=0x7fedd00038a0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 shutdown_connections 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fedbc076480 0x7fedbc078940 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede01413b0 0x7fede01f78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede00cdd90 0x7fede015ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 --2- 172.21.15.163:0/3351255132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede0074020 0x7fede015a850 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 >> 172.21.15.163:0/3351255132 conn(0x7fede006bf40 msgr2=0x7fede015dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:05.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 shutdown_connections 2024-07-26T06:17:05.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.322+0000 7fede7000640 1 -- 172.21.15.163:0/3351255132 wait complete. 2024-07-26T06:17:05.344 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:17:05.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:17:05.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 -- 172.21.15.163:0/1562300596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04003a00 msgr2=0x7f1f04003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:05.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 --2- 172.21.15.163:0/1562300596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04003a00 0x7f1f04003e00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000bb20 tx=0x7f1f04005830 comp rx=0 tx=0).stop 2024-07-26T06:17:05.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 -- 172.21.15.163:0/1562300596 shutdown_connections 2024-07-26T06:17:05.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 --2- 172.21.15.163:0/1562300596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f04003a00 0x7f1f04003e00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 -- 172.21.15.163:0/1562300596 >> 172.21.15.163:0/1562300596 conn(0x7f1f200b4fe0 msgr2=0x7f1f200b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:05.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 -- 172.21.15.163:0/1562300596 shutdown_connections 2024-07-26T06:17:05.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.551+0000 7f1f26200640 1 -- 172.21.15.163:0/1562300596 wait complete. 2024-07-26T06:17:05.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.553+0000 7f1f26200640 1 Processor -- start 2024-07-26T06:17:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.553+0000 7f1f26200640 1 -- start start 2024-07-26T06:17:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f26200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f04003a00 0x7f1f2014d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f26200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 0x7f1f200cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f26200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f200ccc30 0x7f1f2014fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:05.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f26200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f200d2aa0 con 0x7f1f200aaea0 2024-07-26T06:17:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f26200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1f200d2920 con 0x7f1f200ccc30 2024-07-26T06:17:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f26200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1f200d2c20 con 0x7f1f04003a00 2024-07-26T06:17:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 0x7f1f200cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:05.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 0x7f1f200cc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50834/0 (socket says 172.21.15.163:50834) 2024-07-26T06:17:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 -- 172.21.15.163:0/967370770 learned_addr learned my addr 172.21.15.163:0/967370770 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 -- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f04003a00 msgr2=0x7f1f2014d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f1f400640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f04003a00 0x7f1f2014d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f1fe00640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f200ccc30 0x7f1f2014fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f04003a00 0x7f1f2014d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 -- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f200ccc30 msgr2=0x7f1f2014fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f200ccc30 0x7f1f2014fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.554+0000 7f1f17e00640 1 -- 172.21.15.163:0/967370770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f200aa0e0 con 0x7f1f200aaea0 2024-07-26T06:17:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.555+0000 7f1f17e00640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 0x7f1f200cc6f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800d530 tx=0x7f1f0800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.555+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f08019070 con 0x7f1f200aaea0 2024-07-26T06:17:05.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.555+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f080052e0 con 0x7f1f200aaea0 2024-07-26T06:17:05.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.555+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f200aa3c0 con 0x7f1f200aaea0 2024-07-26T06:17:05.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.555+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f08003850 con 0x7f1f200aaea0 2024-07-26T06:17:05.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.555+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f200aa980 con 0x7f1f200aaea0 2024-07-26T06:17:05.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.556+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f200cd040 con 0x7f1f200aaea0 2024-07-26T06:17:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.558+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f1f0801c070 con 0x7f1f200aaea0 2024-07-26T06:17:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.559+0000 7f1f17400640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1efc0764b0 0x7f1efc078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.559+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f1f08097010 con 0x7f1f200aaea0 2024-07-26T06:17:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.559+0000 7f1f1f400640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1efc0764b0 0x7f1efc078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.560+0000 7f1f1f400640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1efc0764b0 0x7f1efc078970 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1f10036000 tx=0x7f1f100046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:05.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.563+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f200cd040 con 0x7f1f200aaea0 2024-07-26T06:17:05.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:05.789+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1f200aab80 con 0x7f1f200aaea0 2024-07-26T06:17:06.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.322+0000 7f1f17400640 1 -- 172.21.15.163:0/967370770 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f1f200aab80 con 0x7f1f200aaea0 2024-07-26T06:17:06.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.326+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1efc0764b0 msgr2=0x7f1efc078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.326+0000 7f1f26200640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1efc0764b0 0x7f1efc078970 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1f10036000 tx=0x7f1f100046c0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.326+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 msgr2=0x7f1f200cc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.326+0000 7f1f26200640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 0x7f1f200cc6f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800d530 tx=0x7f1f0800d9f0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.326+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 shutdown_connections 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.326+0000 7f1f26200640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1efc0764b0 0x7f1efc078970 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.327+0000 7f1f26200640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f200ccc30 0x7f1f2014fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.327+0000 7f1f26200640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f200aaea0 0x7f1f200cc6f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.327+0000 7f1f26200640 1 --2- 172.21.15.163:0/967370770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f04003a00 0x7f1f2014d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.327+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 >> 172.21.15.163:0/967370770 conn(0x7f1f200b4fe0 msgr2=0x7f1f200b6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.327+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 shutdown_connections 2024-07-26T06:17:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.327+0000 7f1f26200640 1 -- 172.21.15.163:0/967370770 wait complete. 2024-07-26T06:17:06.351 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:17:06.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:06.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 -- 172.21.15.163:0/3195901411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 msgr2=0x7f946c0cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 --2- 172.21.15.163:0/3195901411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 0x7f946c0cbd40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9454009830 tx=0x7f945402a510 comp rx=0 tx=0).stop 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 -- 172.21.15.163:0/3195901411 shutdown_connections 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 --2- 172.21.15.163:0/3195901411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 0x7f946c0cbd40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 --2- 172.21.15.163:0/3195901411 >> v2:172.21.15.164:3300/0 conn(0x7f946c146be0 0x7f946c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 -- 172.21.15.163:0/3195901411 >> 172.21.15.163:0/3195901411 conn(0x7f946c0b0e30 msgr2=0x7f946c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 -- 172.21.15.163:0/3195901411 shutdown_connections 2024-07-26T06:17:06.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.555+0000 7f9471400640 1 -- 172.21.15.163:0/3195901411 wait complete. 2024-07-26T06:17:06.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.557+0000 7f9471400640 1 Processor -- start 2024-07-26T06:17:06.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.557+0000 7f9471400640 1 -- start start 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9471400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 0x7f946c13ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9471400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f946c0cc280 0x7f946c13f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9471400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 0x7f946c1f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9471400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0b4970 con 0x7f946c146be0 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9471400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0cc830 con 0x7f946c0cc280 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9471400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f946c0ce520 con 0x7f946c0ade20 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.558+0000 7f9470a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 0x7f946c1f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 0x7f946c1f75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50846/0 (socket says 172.21.15.163:50846) 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 -- 172.21.15.163:0/681554907 learned_addr learned my addr 172.21.15.163:0/681554907 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:06.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9463e00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f946c0cc280 0x7f946c13f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f946be00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 0x7f946c13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 -- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 msgr2=0x7f946c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 0x7f946c13ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 -- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f946c0cc280 msgr2=0x7f946c13f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f946c0cc280 0x7f946c13f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 -- 172.21.15.163:0/681554907 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c13f680 con 0x7f946c146be0 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9463e00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f946c0cc280 0x7f946c13f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:06.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.559+0000 7f9470a00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 0x7f946c1f75b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f946400cbc0 tx=0x7f946400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:06.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.560+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9464016020 con 0x7f946c146be0 2024-07-26T06:17:06.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.560+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f946400add0 con 0x7f946c146be0 2024-07-26T06:17:06.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.560+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9464012680 con 0x7f946c146be0 2024-07-26T06:17:06.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.560+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946c13f950 con 0x7f946c146be0 2024-07-26T06:17:06.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.560+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946c072420 con 0x7f946c146be0 2024-07-26T06:17:06.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.561+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946c072210 con 0x7f946c146be0 2024-07-26T06:17:06.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.562+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f9464019020 con 0x7f946c146be0 2024-07-26T06:17:06.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.563+0000 7f9468c00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9448076480 0x7f9448078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:06.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.564+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9464097ee0 con 0x7f946c146be0 2024-07-26T06:17:06.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.564+0000 7f946be00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9448076480 0x7f9448078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:06.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.565+0000 7f946be00640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9448076480 0x7f9448078940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f945c0099e0 tx=0x7f945c005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:06.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.568+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9464020030 con 0x7f946c146be0 2024-07-26T06:17:06.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.815+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f946c0b0400 con 0x7f946c146be0 2024-07-26T06:17:06.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.816+0000 7f9468c00640 1 -- 172.21.15.163:0/681554907 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f9464061520 con 0x7f946c146be0 2024-07-26T06:17:06.817 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:06.817 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":35,"btime":"2024-07-26T06:17:06:319401+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:02.717555+0000","modified":"2024-07-26T06:17:05.791542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/3071317413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3071317413},{"type":"v1","addr":"172.21.15.164:6835","nonce":3071317413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5053,"qdb_cluster":[5053]},"id":4}]} 2024-07-26T06:17:06.817 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 35 2024-07-26T06:17:06.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.818+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9448076480 msgr2=0x7f9448078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.818+0000 7f9471400640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9448076480 0x7f9448078940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f945c0099e0 tx=0x7f945c005f90 comp rx=0 tx=0).stop 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.818+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 msgr2=0x7f946c1f75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.818+0000 7f9471400640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 0x7f946c1f75b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f946400cbc0 tx=0x7f946400a430 comp rx=0 tx=0).stop 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 shutdown_connections 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9448076480 0x7f9448078940 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f946c146be0 0x7f946c1f75b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f946c0cc280 0x7f946c13f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 --2- 172.21.15.163:0/681554907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f946c0ade20 0x7f946c13ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 >> 172.21.15.163:0/681554907 conn(0x7f946c0b0e30 msgr2=0x7f946c0cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 shutdown_connections 2024-07-26T06:17:06.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:06.819+0000 7f9471400640 1 -- 172.21.15.163:0/681554907 wait complete. 2024-07-26T06:17:06.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.033+0000 7f59e2a00640 1 -- 172.21.15.163:0/2694528274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 msgr2=0x7f59dc0b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.033+0000 7f59e2a00640 1 --2- 172.21.15.163:0/2694528274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc0b2130 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0b1e30 tx=0x7f59d402a8b0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.034+0000 7f59e2a00640 1 -- 172.21.15.163:0/2694528274 shutdown_connections 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.034+0000 7f59e2a00640 1 --2- 172.21.15.163:0/2694528274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc0b2130 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.034+0000 7f59e2a00640 1 --2- 172.21.15.163:0/2694528274 >> v2:172.21.15.164:3300/0 conn(0x7f59dc159370 0x7f59dc159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.034+0000 7f59e2a00640 1 -- 172.21.15.163:0/2694528274 >> 172.21.15.163:0/2694528274 conn(0x7f59dc0b0e30 msgr2=0x7f59dc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.034+0000 7f59e2a00640 1 -- 172.21.15.163:0/2694528274 shutdown_connections 2024-07-26T06:17:07.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.034+0000 7f59e2a00640 1 -- 172.21.15.163:0/2694528274 wait complete. 2024-07-26T06:17:07.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.035+0000 7f59e2a00640 1 Processor -- start 2024-07-26T06:17:07.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.035+0000 7f59e2a00640 1 -- start start 2024-07-26T06:17:07.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.036+0000 7f59e2a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.036+0000 7f59e2a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59dc159370 0x7f59dc147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.036+0000 7f59e2a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59dc0ad8e0 0x7f59dc0add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.036+0000 7f59e2a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc06f2f0 con 0x7f59dc0b4420 2024-07-26T06:17:07.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.036+0000 7f59e2a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc0cd210 con 0x7f59dc0ad8e0 2024-07-26T06:17:07.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.036+0000 7f59e2a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f59dc06f470 con 0x7f59dc159370 2024-07-26T06:17:07.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50870/0 (socket says 172.21.15.163:50870) 2024-07-26T06:17:07.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 -- 172.21.15.163:0/227128269 learned_addr learned my addr 172.21.15.163:0/227128269 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:07.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e0c00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59dc159370 0x7f59dc147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 -- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59dc159370 msgr2=0x7f59dc147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e2000640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59dc0ad8e0 0x7f59dc0add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59dc159370 0x7f59dc147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 -- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59dc0ad8e0 msgr2=0x7f59dc0add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59dc0ad8e0 0x7f59dc0add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 -- 172.21.15.163:0/227128269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc1587c0 con 0x7f59dc0b4420 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.037+0000 7f59e1600640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc158070 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f59d000d740 tx=0x7f59d000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.038+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0017070 con 0x7f59dc0b4420 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.038+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59d0009d20 con 0x7f59dc0b4420 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.038+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc158aa0 con 0x7f59dc0b4420 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.038+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d00106c0 con 0x7f59dc0b4420 2024-07-26T06:17:07.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.038+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc1536c0 con 0x7f59dc0b4420 2024-07-26T06:17:07.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.039+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc071400 con 0x7f59dc0b4420 2024-07-26T06:17:07.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.040+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f59d001a070 con 0x7f59dc0b4420 2024-07-26T06:17:07.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.042+0000 7f59d9600640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59bc076480 0x7f59bc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.042+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f59d0097d90 con 0x7f59dc0b4420 2024-07-26T06:17:07.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.042+0000 7f59e0c00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59bc076480 0x7f59bc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.043+0000 7f59e0c00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59bc076480 0x7f59bc078940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f59dc1f75e0 tx=0x7f59c8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:07.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.046+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f59dc071400 con 0x7f59dc0b4420 2024-07-26T06:17:07.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.251+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59dc0ae730 con 0x7f59dc0b4420 2024-07-26T06:17:07.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.254+0000 7f59d9600640 1 -- 172.21.15.163:0/227128269 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16689 (secure 0 0 0) 0x7f59dc0ae730 con 0x7f59dc0b4420 2024-07-26T06:17:07.255 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:07.273 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":58,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:04.288135+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:00.658125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:01.704071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:07.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59bc076480 msgr2=0x7f59bc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59bc076480 0x7f59bc078940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f59dc1f75e0 tx=0x7f59c8009040 comp rx=0 tx=0).stop 2024-07-26T06:17:07.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 msgr2=0x7f59dc158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc158070 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f59d000d740 tx=0x7f59d000dc00 comp rx=0 tx=0).stop 2024-07-26T06:17:07.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 shutdown_connections 2024-07-26T06:17:07.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59bc076480 0x7f59bc078940 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59dc0ad8e0 0x7f59dc0add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59dc159370 0x7f59dc147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 --2- 172.21.15.163:0/227128269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59dc0b4420 0x7f59dc158070 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 >> 172.21.15.163:0/227128269 conn(0x7f59dc0b0e30 msgr2=0x7f59dc155830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:07.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.257+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 shutdown_connections 2024-07-26T06:17:07.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.258+0000 7f59e2a00640 1 -- 172.21.15.163:0/227128269 wait complete. 2024-07-26T06:17:07.280 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:17:07.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:17:07.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff117200640 1 -- 172.21.15.163:0/2631961157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 msgr2=0x7ff0fc004f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff117200640 1 --2- 172.21.15.163:0/2631961157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff0fc004f60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff108009b50 tx=0x7ff0fc006940 comp rx=0 tx=0).stop 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff116800640 1 -- 172.21.15.163:0/2631961157 reap_dead start 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff117200640 1 -- 172.21.15.163:0/2631961157 shutdown_connections 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff117200640 1 -- 172.21.15.163:0/2631961157 >> 172.21.15.163:0/2631961157 conn(0x7ff1180b0e30 msgr2=0x7ff1180b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff117200640 1 -- 172.21.15.163:0/2631961157 shutdown_connections 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.483+0000 7ff117200640 1 -- 172.21.15.163:0/2631961157 wait complete. 2024-07-26T06:17:07.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.485+0000 7ff117200640 1 Processor -- start 2024-07-26T06:17:07.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.485+0000 7ff117200640 1 -- start start 2024-07-26T06:17:07.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff117200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff1181f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff117200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff118150200 0x7ff118073ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff117200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff11814d540 0x7ff11814d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff117200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181594f0 con 0x7ff0fc004b40 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff117200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff118159370 con 0x7ff11814d540 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff117200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff118159670 con 0x7ff118150200 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff115e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff1181f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff115e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff1181f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50886/0 (socket says 172.21.15.163:50886) 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.486+0000 7ff115e00640 1 -- 172.21.15.163:0/3997875664 learned_addr learned my addr 172.21.15.163:0/3997875664 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff115e00640 1 -- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff118150200 msgr2=0x7ff118073ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff115e00640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff118150200 0x7ff118073ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff115e00640 1 -- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff11814d540 msgr2=0x7ff11814d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff115e00640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff11814d540 0x7ff11814d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff115e00640 1 -- 172.21.15.163:0/3997875664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff118151a30 con 0x7ff0fc004b40 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff115e00640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff1181f47f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff108036040 tx=0x7ff108033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108038040 con 0x7ff0fc004b40 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff108004020 con 0x7ff0fc004b40 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108004470 con 0x7ff0fc004b40 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.487+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff118150990 con 0x7ff0fc004b40 2024-07-26T06:17:07.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.488+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff118150f50 con 0x7ff0fc004b40 2024-07-26T06:17:07.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.489+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11806a1f0 con 0x7ff0fc004b40 2024-07-26T06:17:07.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.490+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7ff108045020 con 0x7ff0fc004b40 2024-07-26T06:17:07.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.491+0000 7ff10e000640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0f4076480 0x7ff0f4078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.492+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7ff1080b8260 con 0x7ff0fc004b40 2024-07-26T06:17:07.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.492+0000 7ff10fe00640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0f4076480 0x7ff0f4078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.493+0000 7ff10fe00640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0f4076480 0x7ff0f4078940 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff100000c00 tx=0x7ff100006900 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:07.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.496+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff11806a1f0 con 0x7ff0fc004b40 2024-07-26T06:17:07.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.701+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1180cbad0 con 0x7ff0f4076480 2024-07-26T06:17:07.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.714+0000 7ff10e000640 1 -- 172.21.15.163:0/3997875664 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+243969 (secure 0 0 0) 0x7ff1180cbad0 con 0x7ff0f4076480 2024-07-26T06:17:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:07.721 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:17:07.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.720+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0f4076480 msgr2=0x7ff0f4078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.720+0000 7ff117200640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0f4076480 0x7ff0f4078940 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff100000c00 tx=0x7ff100006900 comp rx=0 tx=0).stop 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.720+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 msgr2=0x7ff1181f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.720+0000 7ff117200640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff1181f47f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff108036040 tx=0x7ff108033670 comp rx=0 tx=0).stop 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 shutdown_connections 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff0f4076480 0x7ff0f4078940 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff11814d540 0x7ff11814d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff118150200 0x7ff118073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 --2- 172.21.15.163:0/3997875664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff0fc004b40 0x7ff1181f47f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 >> 172.21.15.163:0/3997875664 conn(0x7ff1180b0e30 msgr2=0x7ff1180b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 shutdown_connections 2024-07-26T06:17:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.721+0000 7ff117200640 1 -- 172.21.15.163:0/3997875664 wait complete. 2024-07-26T06:17:07.742 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-07-26T06:17:05.725734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":144,"acting":144,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":234892,"kb_used_data":6212,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754739828,"statfs":{"total":773094113280,"available":772853583872,"internally_reserved":0,"allocated":6361088,"data_stored":3296989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.487474"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.265351+0000","last_change":"2024-07-26T06:17:02.265351+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.265351+0000","last_undegraded":"2024-07-26T06:17:02.265351+0000","last_fullsized":"2024-07-26T06:17:02.265351+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.254791+0000","last_change":"2024-07-26T06:17:02.254791+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.254791+0000","last_undegraded":"2024-07-26T06:17:02.254791+0000","last_fullsized":"2024-07-26T06:17:02.254791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.271195+0000","last_change":"2024-07-26T06:17:02.271195+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.271195+0000","last_undegraded":"2024-07-26T06:17:02.271195+0000","last_fullsized":"2024-07-26T06:17:02.271195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263844+0000","last_change":"2024-07-26T06:17:02.263844+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263844+0000","last_undegraded":"2024-07-26T06:17:02.263844+0000","last_fullsized":"2024-07-26T06:17:02.263844+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.268728+0000","last_change":"2024-07-26T06:17:02.268728+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.268728+0000","last_undegraded":"2024-07-26T06:17:02.268728+0000","last_fullsized":"2024-07-26T06:17:02.268728+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.270623+0000","last_change":"2024-07-26T06:17:02.270623+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.270623+0000","last_undegraded":"2024-07-26T06:17:02.270623+0000","last_fullsized":"2024-07-26T06:17:02.270623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263098+0000","last_change":"2024-07-26T06:17:02.263098+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263098+0000","last_undegraded":"2024-07-26T06:17:02.263098+0000","last_fullsized":"2024-07-26T06:17:02.263098+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.268601+0000","last_change":"2024-07-26T06:17:02.268601+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.268601+0000","last_undegraded":"2024-07-26T06:17:02.268601+0000","last_fullsized":"2024-07-26T06:17:02.268601+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.255189+0000","last_change":"2024-07-26T06:17:02.255189+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.255189+0000","last_undegraded":"2024-07-26T06:17:02.255189+0000","last_fullsized":"2024-07-26T06:17:02.255189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.264050+0000","last_change":"2024-07-26T06:17:02.264050+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.264050+0000","last_undegraded":"2024-07-26T06:17:02.264050+0000","last_fullsized":"2024-07-26T06:17:02.264050+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.255122+0000","last_change":"2024-07-26T06:17:02.255122+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.255122+0000","last_undegraded":"2024-07-26T06:17:02.255122+0000","last_fullsized":"2024-07-26T06:17:02.255122+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.255686+0000","last_change":"2024-07-26T06:17:02.255686+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.255686+0000","last_undegraded":"2024-07-26T06:17:02.255686+0000","last_fullsized":"2024-07-26T06:17:02.255686+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.265366+0000","last_change":"2024-07-26T06:17:02.265366+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.265366+0000","last_undegraded":"2024-07-26T06:17:02.265366+0000","last_fullsized":"2024-07-26T06:17:02.265366+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263129+0000","last_change":"2024-07-26T06:17:02.263129+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263129+0000","last_undegraded":"2024-07-26T06:17:02.263129+0000","last_fullsized":"2024-07-26T06:17:02.263129+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.266389+0000","last_change":"2024-07-26T06:17:02.266389+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.266389+0000","last_undegraded":"2024-07-26T06:17:02.266389+0000","last_fullsized":"2024-07-26T06:17:02.266389+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.274548+0000","last_change":"2024-07-26T06:17:02.274548+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.274548+0000","last_undegraded":"2024-07-26T06:17:02.274548+0000","last_fullsized":"2024-07-26T06:17:02.274548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263097+0000","last_change":"2024-07-26T06:17:02.263097+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263097+0000","last_undegraded":"2024-07-26T06:17:02.263097+0000","last_fullsized":"2024-07-26T06:17:02.263097+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.252346+0000","last_change":"2024-07-26T06:17:02.252346+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.252346+0000","last_undegraded":"2024-07-26T06:17:02.252346+0000","last_fullsized":"2024-07-26T06:17:02.252346+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263078+0000","last_change":"2024-07-26T06:17:02.263078+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263078+0000","last_undegraded":"2024-07-26T06:17:02.263078+0000","last_fullsized":"2024-07-26T06:17:02.263078+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.268069+0000","last_change":"2024-07-26T06:17:02.268069+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.268069+0000","last_undegraded":"2024-07-26T06:17:02.268069+0000","last_fullsized":"2024-07-26T06:17:02.268069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.273613+0000","last_change":"2024-07-26T06:17:02.273613+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.273613+0000","last_undegraded":"2024-07-26T06:17:02.273613+0000","last_fullsized":"2024-07-26T06:17:02.273613+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.288581+0000","last_change":"2024-07-26T06:17:02.287249+0000","last_active":"2024-07-26T06:17:02.288581+0000","last_peered":"2024-07-26T06:17:02.288581+0000","last_clean":"2024-07-26T06:17:02.288581+0000","last_became_active":"2024-07-26T06:17:02.286944+0000","last_became_peered":"2024-07-26T06:17:02.286944+0000","last_unstale":"2024-07-26T06:17:02.288581+0000","last_undegraded":"2024-07-26T06:17:02.288581+0000","last_fullsized":"2024-07-26T06:17:02.288581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:52.941047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.249586+0000","last_change":"2024-07-26T06:17:01.249586+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.249586+0000","last_undegraded":"2024-07-26T06:17:01.249586+0000","last_fullsized":"2024-07-26T06:17:01.249586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.271254+0000","last_change":"2024-07-26T06:17:02.271254+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.271254+0000","last_undegraded":"2024-07-26T06:17:02.271254+0000","last_fullsized":"2024-07-26T06:17:02.271254+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.285478+0000","last_change":"2024-07-26T06:17:02.284029+0000","last_active":"2024-07-26T06:17:02.285478+0000","last_peered":"2024-07-26T06:17:02.285478+0000","last_clean":"2024-07-26T06:17:02.285478+0000","last_became_active":"2024-07-26T06:17:02.283729+0000","last_became_peered":"2024-07-26T06:17:02.283729+0000","last_unstale":"2024-07-26T06:17:02.285478+0000","last_undegraded":"2024-07-26T06:17:02.285478+0000","last_fullsized":"2024-07-26T06:17:02.285478+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:25:34.687692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.274097+0000","last_change":"2024-07-26T06:17:02.274097+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.274097+0000","last_undegraded":"2024-07-26T06:17:02.274097+0000","last_fullsized":"2024-07-26T06:17:02.274097+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.290771+0000","last_change":"2024-07-26T06:17:02.283626+0000","last_active":"2024-07-26T06:17:02.290771+0000","last_peered":"2024-07-26T06:17:02.290771+0000","last_clean":"2024-07-26T06:17:02.290771+0000","last_became_active":"2024-07-26T06:17:02.283345+0000","last_became_peered":"2024-07-26T06:17:02.283345+0000","last_unstale":"2024-07-26T06:17:02.290771+0000","last_undegraded":"2024-07-26T06:17:02.290771+0000","last_fullsized":"2024-07-26T06:17:02.290771+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:04:47.224395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.252557+0000","last_change":"2024-07-26T06:17:01.252557+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.252557+0000","last_undegraded":"2024-07-26T06:17:01.252557+0000","last_fullsized":"2024-07-26T06:17:01.252557+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.285254+0000","last_change":"2024-07-26T06:17:02.282252+0000","last_active":"2024-07-26T06:17:02.285254+0000","last_peered":"2024-07-26T06:17:02.285254+0000","last_clean":"2024-07-26T06:17:02.285254+0000","last_became_active":"2024-07-26T06:17:02.281668+0000","last_became_peered":"2024-07-26T06:17:02.281668+0000","last_unstale":"2024-07-26T06:17:02.285254+0000","last_undegraded":"2024-07-26T06:17:02.285254+0000","last_fullsized":"2024-07-26T06:17:02.285254+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:54.184890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.262798+0000","last_change":"2024-07-26T06:17:02.262798+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.262798+0000","last_undegraded":"2024-07-26T06:17:02.262798+0000","last_fullsized":"2024-07-26T06:17:02.262798+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.290803+0000","last_change":"2024-07-26T06:17:02.283642+0000","last_active":"2024-07-26T06:17:02.290803+0000","last_peered":"2024-07-26T06:17:02.290803+0000","last_clean":"2024-07-26T06:17:02.290803+0000","last_became_active":"2024-07-26T06:17:02.283221+0000","last_became_peered":"2024-07-26T06:17:02.283221+0000","last_unstale":"2024-07-26T06:17:02.290803+0000","last_undegraded":"2024-07-26T06:17:02.290803+0000","last_fullsized":"2024-07-26T06:17:02.290803+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:45.506404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.252733+0000","last_change":"2024-07-26T06:17:01.252733+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.252733+0000","last_undegraded":"2024-07-26T06:17:01.252733+0000","last_fullsized":"2024-07-26T06:17:01.252733+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.282576+0000","last_change":"2024-07-26T06:17:02.279660+0000","last_active":"2024-07-26T06:17:02.282576+0000","last_peered":"2024-07-26T06:17:02.282576+0000","last_clean":"2024-07-26T06:17:02.282576+0000","last_became_active":"2024-07-26T06:17:02.279264+0000","last_became_peered":"2024-07-26T06:17:02.279264+0000","last_unstale":"2024-07-26T06:17:02.282576+0000","last_undegraded":"2024-07-26T06:17:02.282576+0000","last_fullsized":"2024-07-26T06:17:02.282576+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:09.935590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.252098+0000","last_change":"2024-07-26T06:17:02.252098+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.252098+0000","last_undegraded":"2024-07-26T06:17:02.252098+0000","last_fullsized":"2024-07-26T06:17:02.252098+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.248955+0000","last_change":"2024-07-26T06:17:01.248955+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.248955+0000","last_undegraded":"2024-07-26T06:17:01.248955+0000","last_fullsized":"2024-07-26T06:17:01.248955+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.280299+0000","last_change":"2024-07-26T06:17:02.275540+0000","last_active":"2024-07-26T06:17:02.280299+0000","last_peered":"2024-07-26T06:17:02.280299+0000","last_clean":"2024-07-26T06:17:02.280299+0000","last_became_active":"2024-07-26T06:17:02.275124+0000","last_became_peered":"2024-07-26T06:17:02.275124+0000","last_unstale":"2024-07-26T06:17:02.280299+0000","last_undegraded":"2024-07-26T06:17:02.280299+0000","last_fullsized":"2024-07-26T06:17:02.280299+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:45.506404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.253686+0000","last_change":"2024-07-26T06:17:02.253686+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.253686+0000","last_undegraded":"2024-07-26T06:17:02.253686+0000","last_fullsized":"2024-07-26T06:17:02.253686+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.254036+0000","last_change":"2024-07-26T06:17:01.254036+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.254036+0000","last_undegraded":"2024-07-26T06:17:01.254036+0000","last_fullsized":"2024-07-26T06:17:01.254036+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.270270+0000","last_change":"2024-07-26T06:17:02.270270+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.270270+0000","last_undegraded":"2024-07-26T06:17:02.270270+0000","last_fullsized":"2024-07-26T06:17:02.270270+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.249550+0000","last_change":"2024-07-26T06:17:01.249550+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.249550+0000","last_undegraded":"2024-07-26T06:17:01.249550+0000","last_fullsized":"2024-07-26T06:17:01.249550+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.253314+0000","last_change":"2024-07-26T06:17:02.253314+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.253314+0000","last_undegraded":"2024-07-26T06:17:02.253314+0000","last_fullsized":"2024-07-26T06:17:02.253314+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.286330+0000","last_change":"2024-07-26T06:17:02.278586+0000","last_active":"2024-07-26T06:17:02.286330+0000","last_peered":"2024-07-26T06:17:02.286330+0000","last_clean":"2024-07-26T06:17:02.286330+0000","last_became_active":"2024-07-26T06:17:02.278327+0000","last_became_peered":"2024-07-26T06:17:02.278327+0000","last_unstale":"2024-07-26T06:17:02.286330+0000","last_undegraded":"2024-07-26T06:17:02.286330+0000","last_fullsized":"2024-07-26T06:17:02.286330+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:19:19.944799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.284736+0000","last_change":"2024-07-26T06:17:02.278462+0000","last_active":"2024-07-26T06:17:02.284736+0000","last_peered":"2024-07-26T06:17:02.284736+0000","last_clean":"2024-07-26T06:17:02.284736+0000","last_became_active":"2024-07-26T06:17:02.278115+0000","last_became_peered":"2024-07-26T06:17:02.278115+0000","last_unstale":"2024-07-26T06:17:02.284736+0000","last_undegraded":"2024-07-26T06:17:02.284736+0000","last_fullsized":"2024-07-26T06:17:02.284736+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:18.119573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.251887+0000","last_change":"2024-07-26T06:17:02.251887+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.251887+0000","last_undegraded":"2024-07-26T06:17:02.251887+0000","last_fullsized":"2024-07-26T06:17:02.251887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.286948+0000","last_change":"2024-07-26T06:17:02.285212+0000","last_active":"2024-07-26T06:17:02.286948+0000","last_peered":"2024-07-26T06:17:02.286948+0000","last_clean":"2024-07-26T06:17:02.286948+0000","last_became_active":"2024-07-26T06:17:02.284781+0000","last_became_peered":"2024-07-26T06:17:02.284781+0000","last_unstale":"2024-07-26T06:17:02.286948+0000","last_undegraded":"2024-07-26T06:17:02.286948+0000","last_fullsized":"2024-07-26T06:17:02.286948+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:42.996320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.252060+0000","last_change":"2024-07-26T06:17:02.252060+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.252060+0000","last_undegraded":"2024-07-26T06:17:02.252060+0000","last_fullsized":"2024-07-26T06:17:02.252060+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.249539+0000","last_change":"2024-07-26T06:17:01.249539+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.249539+0000","last_undegraded":"2024-07-26T06:17:01.249539+0000","last_fullsized":"2024-07-26T06:17:01.249539+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.271713+0000","last_change":"2024-07-26T06:17:02.271713+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.271713+0000","last_undegraded":"2024-07-26T06:17:02.271713+0000","last_fullsized":"2024-07-26T06:17:02.271713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.282382+0000","last_change":"2024-07-26T06:17:02.280600+0000","last_active":"2024-07-26T06:17:02.282382+0000","last_peered":"2024-07-26T06:17:02.282382+0000","last_clean":"2024-07-26T06:17:02.282382+0000","last_became_active":"2024-07-26T06:17:02.280312+0000","last_became_peered":"2024-07-26T06:17:02.280312+0000","last_unstale":"2024-07-26T06:17:02.282382+0000","last_undegraded":"2024-07-26T06:17:02.282382+0000","last_fullsized":"2024-07-26T06:17:02.282382+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:32:07.106668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.286823+0000","last_change":"2024-07-26T06:17:02.282104+0000","last_active":"2024-07-26T06:17:02.286823+0000","last_peered":"2024-07-26T06:17:02.286823+0000","last_clean":"2024-07-26T06:17:02.286823+0000","last_became_active":"2024-07-26T06:17:02.281805+0000","last_became_peered":"2024-07-26T06:17:02.281805+0000","last_unstale":"2024-07-26T06:17:02.286823+0000","last_undegraded":"2024-07-26T06:17:02.286823+0000","last_fullsized":"2024-07-26T06:17:02.286823+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:52.941047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.252055+0000","last_change":"2024-07-26T06:17:02.252055+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.252055+0000","last_undegraded":"2024-07-26T06:17:02.252055+0000","last_fullsized":"2024-07-26T06:17:02.252055+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.282341+0000","last_change":"2024-07-26T06:17:02.279333+0000","last_active":"2024-07-26T06:17:02.282341+0000","last_peered":"2024-07-26T06:17:02.282341+0000","last_clean":"2024-07-26T06:17:02.282341+0000","last_became_active":"2024-07-26T06:17:02.279019+0000","last_became_peered":"2024-07-26T06:17:02.279019+0000","last_unstale":"2024-07-26T06:17:02.282341+0000","last_undegraded":"2024-07-26T06:17:02.282341+0000","last_fullsized":"2024-07-26T06:17:02.282341+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:48:49.951209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263933+0000","last_change":"2024-07-26T06:17:02.263933+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263933+0000","last_undegraded":"2024-07-26T06:17:02.263933+0000","last_fullsized":"2024-07-26T06:17:02.263933+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.248959+0000","last_change":"2024-07-26T06:17:01.248959+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.248959+0000","last_undegraded":"2024-07-26T06:17:01.248959+0000","last_fullsized":"2024-07-26T06:17:01.248959+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.269885+0000","last_change":"2024-07-26T06:17:02.269885+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.269885+0000","last_undegraded":"2024-07-26T06:17:02.269885+0000","last_fullsized":"2024-07-26T06:17:02.269885+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.250359+0000","last_change":"2024-07-26T06:17:01.250359+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.250359+0000","last_undegraded":"2024-07-26T06:17:01.250359+0000","last_fullsized":"2024-07-26T06:17:01.250359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.268499+0000","last_change":"2024-07-26T06:17:02.268499+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.268499+0000","last_undegraded":"2024-07-26T06:17:02.268499+0000","last_fullsized":"2024-07-26T06:17:02.268499+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.290811+0000","last_change":"2024-07-26T06:17:02.284878+0000","last_active":"2024-07-26T06:17:02.290811+0000","last_peered":"2024-07-26T06:17:02.290811+0000","last_clean":"2024-07-26T06:17:02.290811+0000","last_became_active":"2024-07-26T06:17:02.284572+0000","last_became_peered":"2024-07-26T06:17:02.284572+0000","last_unstale":"2024-07-26T06:17:02.290811+0000","last_undegraded":"2024-07-26T06:17:02.290811+0000","last_fullsized":"2024-07-26T06:17:02.290811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:09.935590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.263339+0000","last_change":"2024-07-26T06:17:02.263339+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.263339+0000","last_undegraded":"2024-07-26T06:17:02.263339+0000","last_fullsized":"2024-07-26T06:17:02.263339+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.250957+0000","last_change":"2024-07-26T06:17:01.250957+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.250957+0000","last_undegraded":"2024-07-26T06:17:01.250957+0000","last_fullsized":"2024-07-26T06:17:01.250957+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.269030+0000","last_change":"2024-07-26T06:17:02.269030+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.269030+0000","last_undegraded":"2024-07-26T06:17:02.269030+0000","last_fullsized":"2024-07-26T06:17:02.269030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.288553+0000","last_change":"2024-07-26T06:17:02.286291+0000","last_active":"2024-07-26T06:17:02.288553+0000","last_peered":"2024-07-26T06:17:02.288553+0000","last_clean":"2024-07-26T06:17:02.288553+0000","last_became_active":"2024-07-26T06:17:02.285956+0000","last_became_peered":"2024-07-26T06:17:02.285956+0000","last_unstale":"2024-07-26T06:17:02.288553+0000","last_undegraded":"2024-07-26T06:17:02.288553+0000","last_fullsized":"2024-07-26T06:17:02.288553+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:18.119573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.286417+0000","last_change":"2024-07-26T06:17:02.283880+0000","last_active":"2024-07-26T06:17:02.286417+0000","last_peered":"2024-07-26T06:17:02.286417+0000","last_clean":"2024-07-26T06:17:02.286417+0000","last_became_active":"2024-07-26T06:17:02.281865+0000","last_became_peered":"2024-07-26T06:17:02.281865+0000","last_unstale":"2024-07-26T06:17:02.286417+0000","last_undegraded":"2024-07-26T06:17:02.286417+0000","last_fullsized":"2024-07-26T06:17:02.286417+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:25.982989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.273572+0000","last_change":"2024-07-26T06:17:02.273572+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.273572+0000","last_undegraded":"2024-07-26T06:17:02.273572+0000","last_fullsized":"2024-07-26T06:17:02.273572+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.294009+0000","last_change":"2024-07-26T06:17:02.288315+0000","last_active":"2024-07-26T06:17:02.294009+0000","last_peered":"2024-07-26T06:17:02.294009+0000","last_clean":"2024-07-26T06:17:02.294009+0000","last_became_active":"2024-07-26T06:17:02.287947+0000","last_became_peered":"2024-07-26T06:17:02.287947+0000","last_unstale":"2024-07-26T06:17:02.294009+0000","last_undegraded":"2024-07-26T06:17:02.294009+0000","last_fullsized":"2024-07-26T06:17:02.294009+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:42.996320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.251058+0000","last_change":"2024-07-26T06:17:01.251058+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.251058+0000","last_undegraded":"2024-07-26T06:17:01.251058+0000","last_fullsized":"2024-07-26T06:17:01.251058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-07-26T06:17:02.252163+0000","last_change":"2024-07-26T06:17:02.252163+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.252163+0000","last_undegraded":"2024-07-26T06:17:02.252163+0000","last_fullsized":"2024-07-26T06:17:02.252163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.254048+0000","last_change":"2024-07-26T06:17:01.254048+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.254048+0000","last_undegraded":"2024-07-26T06:17:01.254048+0000","last_fullsized":"2024-07-26T06:17:01.254048+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.249529+0000","last_change":"2024-07-26T06:17:01.249529+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.249529+0000","last_undegraded":"2024-07-26T06:17:01.249529+0000","last_fullsized":"2024-07-26T06:17:01.249529+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-07-26T06:17:01.248374+0000","last_change":"2024-07-26T06:17:01.248374+0000","last_active":"2024-07-26T06:17:01.232108+0000","last_peered":"2024-07-26T06:17:01.232108+0000","last_clean":"2024-07-26T06:17:01.232108+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:01.248374+0000","last_undegraded":"2024-07-26T06:17:01.248374+0000","last_fullsized":"2024-07-26T06:17:01.248374+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:17:02.236608+0000","last_change":"2024-07-26T06:17:02.236608+0000","last_active":"2024-07-26T06:17:02.236608+0000","last_peered":"2024-07-26T06:17:02.236608+0000","last_clean":"2024-07-26T06:17:02.236608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:02.236608+0000","last_undegraded":"2024-07-26T06:17:02.236608+0000","last_fullsized":"2024-07-26T06:17:02.236608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.287944+0000","last_change":"2024-07-26T06:17:02.285386+0000","last_active":"2024-07-26T06:17:02.287944+0000","last_peered":"2024-07-26T06:17:02.287944+0000","last_clean":"2024-07-26T06:17:02.287944+0000","last_became_active":"2024-07-26T06:17:02.285055+0000","last_became_peered":"2024-07-26T06:17:02.285055+0000","last_unstale":"2024-07-26T06:17:02.287944+0000","last_undegraded":"2024-07-26T06:17:02.287944+0000","last_fullsized":"2024-07-26T06:17:02.287944+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:08:43.737257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":173,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-07-26T06:17:02.258772+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:17:02.258772+0000","last_peered":"2024-07-26T06:17:02.258772+0000","last_clean":"2024-07-26T06:17:02.258772+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:17:02.258772+0000","last_undegraded":"2024-07-26T06:17:02.258772+0000","last_fullsized":"2024-07-26T06:17:02.258772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":5},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1245184,"data_stored":857081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509488,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":266713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509487,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":266713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":266713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509488,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1245184,"data_stored":857081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":638976,"data_stored":260896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":638976,"data_stored":260896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542192,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":638976,"data_stored":260896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:17:07.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:17:07.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 -- 172.21.15.163:0/463222186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 msgr2=0x7f37d80b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 --2- 172.21.15.163:0/463222186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d80b4bd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005fd0 tx=0x7f37cc02a4a0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 -- 172.21.15.163:0/463222186 shutdown_connections 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 --2- 172.21.15.163:0/463222186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d80ae870 0x7f37d80b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 --2- 172.21.15.163:0/463222186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d80b4bd0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 -- 172.21.15.163:0/463222186 >> 172.21.15.163:0/463222186 conn(0x7f37d80b0e30 msgr2=0x7f37d80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 -- 172.21.15.163:0/463222186 shutdown_connections 2024-07-26T06:17:07.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.989+0000 7f37df000640 1 -- 172.21.15.163:0/463222186 wait complete. 2024-07-26T06:17:07.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.991+0000 7f37df000640 1 Processor -- start 2024-07-26T06:17:07.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.991+0000 7f37df000640 1 -- start start 2024-07-26T06:17:07.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.992+0000 7f37df000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37d80ae870 0x7f37d81f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.992+0000 7f37df000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d815cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.992+0000 7f37df000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d815db10 0x7f37d8152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.992+0000 7f37df000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37d80cb870 con 0x7f37d80b3bd0 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.992+0000 7f37df000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f37d8073300 con 0x7f37d80ae870 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.992+0000 7f37df000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f37d80cb9f0 con 0x7f37d815db10 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d815cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d815cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50900/0 (socket says 172.21.15.163:50900) 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 -- 172.21.15.163:0/2887962410 learned_addr learned my addr 172.21.15.163:0/2887962410 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 -- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d815db10 msgr2=0x7f37d8152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37de600640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d815db10 0x7f37d8152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37ddc00640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37d80ae870 0x7f37d81f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d815db10 0x7f37d8152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 -- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37d80ae870 msgr2=0x7f37d81f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37d80ae870 0x7f37d81f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 -- 172.21.15.163:0/2887962410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d8150e60 con 0x7f37d80b3bd0 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37de600640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d815db10 0x7f37d8152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.993+0000 7f37dd200640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d815cff0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f37c800c9a0 tx=0x7f37c800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.994+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8007bb0 con 0x7f37d80b3bd0 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.994+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37c8005020 con 0x7f37d80b3bd0 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.994+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8007e60 con 0x7f37d80b3bd0 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.994+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d8151140 con 0x7f37d80b3bd0 2024-07-26T06:17:07.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.994+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d8151700 con 0x7f37d80b3bd0 2024-07-26T06:17:07.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.995+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d80b3fb0 con 0x7f37d80b3bd0 2024-07-26T06:17:07.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.996+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f37c8005470 con 0x7f37d80b3bd0 2024-07-26T06:17:07.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.998+0000 7f37d5600640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f37bc076480 0x7f37bc078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:08.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.999+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f37c8098110 con 0x7f37d80b3bd0 2024-07-26T06:17:08.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:07.999+0000 7f37ddc00640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f37bc076480 0x7f37bc078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:08.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.000+0000 7f37ddc00640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f37bc076480 0x7f37bc078940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f37cc033000 tx=0x7f37cc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:08.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.003+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f37c8014030 con 0x7f37d80b3bd0 2024-07-26T06:17:08.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.206+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37d80cbce0 con 0x7f37bc076480 2024-07-26T06:17:08.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.220+0000 7f37d5600640 1 -- 172.21.15.163:0/2887962410 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7f37d80cbce0 con 0x7f37bc076480 2024-07-26T06:17:08.225 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:08.227 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:17:08.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f37bc076480 msgr2=0x7f37bc078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f37bc076480 0x7f37bc078940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f37cc033000 tx=0x7f37cc004730 comp rx=0 tx=0).stop 2024-07-26T06:17:08.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 msgr2=0x7f37d815cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d815cff0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f37c800c9a0 tx=0x7f37c800ce60 comp rx=0 tx=0).stop 2024-07-26T06:17:08.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 shutdown_connections 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f37bc076480 0x7f37bc078940 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37d815db10 0x7f37d8152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f37d80b3bd0 0x7f37d815cff0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 --2- 172.21.15.163:0/2887962410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37d80ae870 0x7f37d81f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 >> 172.21.15.163:0/2887962410 conn(0x7f37d80b0e30 msgr2=0x7f37d80adc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 shutdown_connections 2024-07-26T06:17:08.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.227+0000 7f37df000640 1 -- 172.21.15.163:0/2887962410 wait complete. 2024-07-26T06:17:08.248 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-07-26T06:17:07.726370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239448,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735272,"statfs":{"total":773094113280,"available":772848918528,"internally_reserved":0,"allocated":6758400,"data_stored":3473996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001946"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.771450+0000","last_change":"2024-07-26T06:17:03.289893+0000","last_active":"2024-07-26T06:17:04.771450+0000","last_peered":"2024-07-26T06:17:04.771450+0000","last_clean":"2024-07-26T06:17:04.771450+0000","last_became_active":"2024-07-26T06:17:03.289514+0000","last_became_peered":"2024-07-26T06:17:03.289514+0000","last_unstale":"2024-07-26T06:17:04.771450+0000","last_undegraded":"2024-07-26T06:17:04.771450+0000","last_fullsized":"2024-07-26T06:17:04.771450+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:34:46.162573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301233+0000","last_change":"2024-07-26T06:17:03.316406+0000","last_active":"2024-07-26T06:17:04.301233+0000","last_peered":"2024-07-26T06:17:04.301233+0000","last_clean":"2024-07-26T06:17:04.301233+0000","last_became_active":"2024-07-26T06:17:03.312123+0000","last_became_peered":"2024-07-26T06:17:03.312123+0000","last_unstale":"2024-07-26T06:17:04.301233+0000","last_undegraded":"2024-07-26T06:17:04.301233+0000","last_fullsized":"2024-07-26T06:17:04.301233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:33:28.082240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709808+0000","last_change":"2024-07-26T06:17:03.303052+0000","last_active":"2024-07-26T06:17:04.709808+0000","last_peered":"2024-07-26T06:17:04.709808+0000","last_clean":"2024-07-26T06:17:04.709808+0000","last_became_active":"2024-07-26T06:17:03.301564+0000","last_became_peered":"2024-07-26T06:17:03.301564+0000","last_unstale":"2024-07-26T06:17:04.709808+0000","last_undegraded":"2024-07-26T06:17:04.709808+0000","last_fullsized":"2024-07-26T06:17:04.709808+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:25:43.511989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034828+0000","last_change":"2024-07-26T06:17:03.303257+0000","last_active":"2024-07-26T06:17:05.034828+0000","last_peered":"2024-07-26T06:17:05.034828+0000","last_clean":"2024-07-26T06:17:05.034828+0000","last_became_active":"2024-07-26T06:17:03.302955+0000","last_became_peered":"2024-07-26T06:17:03.302955+0000","last_unstale":"2024-07-26T06:17:05.034828+0000","last_undegraded":"2024-07-26T06:17:05.034828+0000","last_fullsized":"2024-07-26T06:17:05.034828+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:27.548776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.771584+0000","last_change":"2024-07-26T06:17:03.299106+0000","last_active":"2024-07-26T06:17:04.771584+0000","last_peered":"2024-07-26T06:17:04.771584+0000","last_clean":"2024-07-26T06:17:04.771584+0000","last_became_active":"2024-07-26T06:17:03.298838+0000","last_became_peered":"2024-07-26T06:17:03.298838+0000","last_unstale":"2024-07-26T06:17:04.771584+0000","last_undegraded":"2024-07-26T06:17:04.771584+0000","last_fullsized":"2024-07-26T06:17:04.771584+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:21:29.949839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032972+0000","last_change":"2024-07-26T06:17:03.321159+0000","last_active":"2024-07-26T06:17:05.032972+0000","last_peered":"2024-07-26T06:17:05.032972+0000","last_clean":"2024-07-26T06:17:05.032972+0000","last_became_active":"2024-07-26T06:17:03.320344+0000","last_became_peered":"2024-07-26T06:17:03.320344+0000","last_unstale":"2024-07-26T06:17:05.032972+0000","last_undegraded":"2024-07-26T06:17:05.032972+0000","last_fullsized":"2024-07-26T06:17:05.032972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:33.919546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299645+0000","last_change":"2024-07-26T06:17:03.301083+0000","last_active":"2024-07-26T06:17:04.299645+0000","last_peered":"2024-07-26T06:17:04.299645+0000","last_clean":"2024-07-26T06:17:04.299645+0000","last_became_active":"2024-07-26T06:17:03.300110+0000","last_became_peered":"2024-07-26T06:17:03.300110+0000","last_unstale":"2024-07-26T06:17:04.299645+0000","last_undegraded":"2024-07-26T06:17:04.299645+0000","last_fullsized":"2024-07-26T06:17:04.299645+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:12:16.563259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301269+0000","last_change":"2024-07-26T06:17:03.295535+0000","last_active":"2024-07-26T06:17:04.301269+0000","last_peered":"2024-07-26T06:17:04.301269+0000","last_clean":"2024-07-26T06:17:04.301269+0000","last_became_active":"2024-07-26T06:17:03.295193+0000","last_became_peered":"2024-07-26T06:17:03.295193+0000","last_unstale":"2024-07-26T06:17:04.301269+0000","last_undegraded":"2024-07-26T06:17:04.301269+0000","last_fullsized":"2024-07-26T06:17:04.301269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:25.584990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300967+0000","last_change":"2024-07-26T06:17:03.321779+0000","last_active":"2024-07-26T06:17:04.300967+0000","last_peered":"2024-07-26T06:17:04.300967+0000","last_clean":"2024-07-26T06:17:04.300967+0000","last_became_active":"2024-07-26T06:17:03.321504+0000","last_became_peered":"2024-07-26T06:17:03.321504+0000","last_unstale":"2024-07-26T06:17:04.300967+0000","last_undegraded":"2024-07-26T06:17:04.300967+0000","last_fullsized":"2024-07-26T06:17:04.300967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:09.404388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299631+0000","last_change":"2024-07-26T06:17:03.304523+0000","last_active":"2024-07-26T06:17:04.299631+0000","last_peered":"2024-07-26T06:17:04.299631+0000","last_clean":"2024-07-26T06:17:04.299631+0000","last_became_active":"2024-07-26T06:17:03.304201+0000","last_became_peered":"2024-07-26T06:17:03.304201+0000","last_unstale":"2024-07-26T06:17:04.299631+0000","last_undegraded":"2024-07-26T06:17:04.299631+0000","last_fullsized":"2024-07-26T06:17:04.299631+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:25:32.314901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300415+0000","last_change":"2024-07-26T06:17:03.311501+0000","last_active":"2024-07-26T06:17:04.300415+0000","last_peered":"2024-07-26T06:17:04.300415+0000","last_clean":"2024-07-26T06:17:04.300415+0000","last_became_active":"2024-07-26T06:17:03.310985+0000","last_became_peered":"2024-07-26T06:17:03.310985+0000","last_unstale":"2024-07-26T06:17:04.300415+0000","last_undegraded":"2024-07-26T06:17:04.300415+0000","last_fullsized":"2024-07-26T06:17:04.300415+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:43:33.481335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034887+0000","last_change":"2024-07-26T06:17:03.312236+0000","last_active":"2024-07-26T06:17:05.034887+0000","last_peered":"2024-07-26T06:17:05.034887+0000","last_clean":"2024-07-26T06:17:05.034887+0000","last_became_active":"2024-07-26T06:17:03.311753+0000","last_became_peered":"2024-07-26T06:17:03.311753+0000","last_unstale":"2024-07-26T06:17:05.034887+0000","last_undegraded":"2024-07-26T06:17:05.034887+0000","last_fullsized":"2024-07-26T06:17:05.034887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:33.919546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033331+0000","last_change":"2024-07-26T06:17:03.314191+0000","last_active":"2024-07-26T06:17:05.033331+0000","last_peered":"2024-07-26T06:17:05.033331+0000","last_clean":"2024-07-26T06:17:05.033331+0000","last_became_active":"2024-07-26T06:17:03.313896+0000","last_became_peered":"2024-07-26T06:17:03.313896+0000","last_unstale":"2024-07-26T06:17:05.033331+0000","last_undegraded":"2024-07-26T06:17:05.033331+0000","last_fullsized":"2024-07-26T06:17:05.033331+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:32:08.111167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300187+0000","last_change":"2024-07-26T06:17:03.283111+0000","last_active":"2024-07-26T06:17:04.300187+0000","last_peered":"2024-07-26T06:17:04.300187+0000","last_clean":"2024-07-26T06:17:04.300187+0000","last_became_active":"2024-07-26T06:17:03.282812+0000","last_became_peered":"2024-07-26T06:17:03.282812+0000","last_unstale":"2024-07-26T06:17:04.300187+0000","last_undegraded":"2024-07-26T06:17:04.300187+0000","last_fullsized":"2024-07-26T06:17:04.300187+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:12.506797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300568+0000","last_change":"2024-07-26T06:17:03.309871+0000","last_active":"2024-07-26T06:17:04.300568+0000","last_peered":"2024-07-26T06:17:04.300568+0000","last_clean":"2024-07-26T06:17:04.300568+0000","last_became_active":"2024-07-26T06:17:03.307700+0000","last_became_peered":"2024-07-26T06:17:03.307700+0000","last_unstale":"2024-07-26T06:17:04.300568+0000","last_undegraded":"2024-07-26T06:17:04.300568+0000","last_fullsized":"2024-07-26T06:17:04.300568+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:23:13.020686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032745+0000","last_change":"2024-07-26T06:17:03.313879+0000","last_active":"2024-07-26T06:17:05.032745+0000","last_peered":"2024-07-26T06:17:05.032745+0000","last_clean":"2024-07-26T06:17:05.032745+0000","last_became_active":"2024-07-26T06:17:03.313517+0000","last_became_peered":"2024-07-26T06:17:03.313517+0000","last_unstale":"2024-07-26T06:17:05.032745+0000","last_undegraded":"2024-07-26T06:17:05.032745+0000","last_fullsized":"2024-07-26T06:17:05.032745+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:50:56.277291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300293+0000","last_change":"2024-07-26T06:17:03.295390+0000","last_active":"2024-07-26T06:17:04.300293+0000","last_peered":"2024-07-26T06:17:04.300293+0000","last_clean":"2024-07-26T06:17:04.300293+0000","last_became_active":"2024-07-26T06:17:03.294958+0000","last_became_peered":"2024-07-26T06:17:03.294958+0000","last_unstale":"2024-07-26T06:17:04.300293+0000","last_undegraded":"2024-07-26T06:17:04.300293+0000","last_fullsized":"2024-07-26T06:17:04.300293+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:28:39.157948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710111+0000","last_change":"2024-07-26T06:17:03.317903+0000","last_active":"2024-07-26T06:17:04.710111+0000","last_peered":"2024-07-26T06:17:04.710111+0000","last_clean":"2024-07-26T06:17:04.710111+0000","last_became_active":"2024-07-26T06:17:03.317628+0000","last_became_peered":"2024-07-26T06:17:03.317628+0000","last_unstale":"2024-07-26T06:17:04.710111+0000","last_undegraded":"2024-07-26T06:17:04.710111+0000","last_fullsized":"2024-07-26T06:17:04.710111+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:20:50.310800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.771316+0000","last_change":"2024-07-26T06:17:03.292635+0000","last_active":"2024-07-26T06:17:04.771316+0000","last_peered":"2024-07-26T06:17:04.771316+0000","last_clean":"2024-07-26T06:17:04.771316+0000","last_became_active":"2024-07-26T06:17:03.292324+0000","last_became_peered":"2024-07-26T06:17:03.292324+0000","last_unstale":"2024-07-26T06:17:04.771316+0000","last_undegraded":"2024-07-26T06:17:04.771316+0000","last_fullsized":"2024-07-26T06:17:04.771316+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:26.753625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300802+0000","last_change":"2024-07-26T06:17:03.315199+0000","last_active":"2024-07-26T06:17:04.300802+0000","last_peered":"2024-07-26T06:17:04.300802+0000","last_clean":"2024-07-26T06:17:04.300802+0000","last_became_active":"2024-07-26T06:17:03.314905+0000","last_became_peered":"2024-07-26T06:17:03.314905+0000","last_unstale":"2024-07-26T06:17:04.300802+0000","last_undegraded":"2024-07-26T06:17:04.300802+0000","last_fullsized":"2024-07-26T06:17:04.300802+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:49.283977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709406+0000","last_change":"2024-07-26T06:17:03.315373+0000","last_active":"2024-07-26T06:17:04.709406+0000","last_peered":"2024-07-26T06:17:04.709406+0000","last_clean":"2024-07-26T06:17:04.709406+0000","last_became_active":"2024-07-26T06:17:03.314647+0000","last_became_peered":"2024-07-26T06:17:03.314647+0000","last_unstale":"2024-07-26T06:17:04.709406+0000","last_undegraded":"2024-07-26T06:17:04.709406+0000","last_fullsized":"2024-07-26T06:17:04.709406+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:59.791876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299770+0000","last_change":"2024-07-26T06:17:03.303196+0000","last_active":"2024-07-26T06:17:04.299770+0000","last_peered":"2024-07-26T06:17:04.299770+0000","last_clean":"2024-07-26T06:17:04.299770+0000","last_became_active":"2024-07-26T06:17:03.299479+0000","last_became_peered":"2024-07-26T06:17:03.299479+0000","last_unstale":"2024-07-26T06:17:04.299770+0000","last_undegraded":"2024-07-26T06:17:04.299770+0000","last_fullsized":"2024-07-26T06:17:04.299770+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:41:16.966877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709348+0000","last_change":"2024-07-26T06:17:03.304649+0000","last_active":"2024-07-26T06:17:04.709348+0000","last_peered":"2024-07-26T06:17:04.709348+0000","last_clean":"2024-07-26T06:17:04.709348+0000","last_became_active":"2024-07-26T06:17:03.303387+0000","last_became_peered":"2024-07-26T06:17:03.303387+0000","last_unstale":"2024-07-26T06:17:04.709348+0000","last_undegraded":"2024-07-26T06:17:04.709348+0000","last_fullsized":"2024-07-26T06:17:04.709348+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:27.548776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032987+0000","last_change":"2024-07-26T06:17:03.319978+0000","last_active":"2024-07-26T06:17:05.032987+0000","last_peered":"2024-07-26T06:17:05.032987+0000","last_clean":"2024-07-26T06:17:05.032987+0000","last_became_active":"2024-07-26T06:17:03.319456+0000","last_became_peered":"2024-07-26T06:17:03.319456+0000","last_unstale":"2024-07-26T06:17:05.032987+0000","last_undegraded":"2024-07-26T06:17:05.032987+0000","last_fullsized":"2024-07-26T06:17:05.032987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:48:50.955708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.771574+0000","last_change":"2024-07-26T06:17:03.306343+0000","last_active":"2024-07-26T06:17:04.771574+0000","last_peered":"2024-07-26T06:17:04.771574+0000","last_clean":"2024-07-26T06:17:04.771574+0000","last_became_active":"2024-07-26T06:17:03.306008+0000","last_became_peered":"2024-07-26T06:17:03.306008+0000","last_unstale":"2024-07-26T06:17:04.771574+0000","last_undegraded":"2024-07-26T06:17:04.771574+0000","last_fullsized":"2024-07-26T06:17:04.771574+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:18:31.539873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035563+0000","last_change":"2024-07-26T06:17:03.297443+0000","last_active":"2024-07-26T06:17:05.035563+0000","last_peered":"2024-07-26T06:17:05.035563+0000","last_clean":"2024-07-26T06:17:05.035563+0000","last_became_active":"2024-07-26T06:17:03.297149+0000","last_became_peered":"2024-07-26T06:17:03.297149+0000","last_unstale":"2024-07-26T06:17:05.035563+0000","last_undegraded":"2024-07-26T06:17:05.035563+0000","last_fullsized":"2024-07-26T06:17:05.035563+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:25:43.511989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033205+0000","last_change":"2024-07-26T06:17:03.321627+0000","last_active":"2024-07-26T06:17:05.033205+0000","last_peered":"2024-07-26T06:17:05.033205+0000","last_clean":"2024-07-26T06:17:05.033205+0000","last_became_active":"2024-07-26T06:17:03.320933+0000","last_became_peered":"2024-07-26T06:17:03.320933+0000","last_unstale":"2024-07-26T06:17:05.033205+0000","last_undegraded":"2024-07-26T06:17:05.033205+0000","last_fullsized":"2024-07-26T06:17:05.033205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:21.198904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.771350+0000","last_change":"2024-07-26T06:17:03.301931+0000","last_active":"2024-07-26T06:17:04.771350+0000","last_peered":"2024-07-26T06:17:04.771350+0000","last_clean":"2024-07-26T06:17:04.771350+0000","last_became_active":"2024-07-26T06:17:03.300977+0000","last_became_peered":"2024-07-26T06:17:03.300977+0000","last_unstale":"2024-07-26T06:17:04.771350+0000","last_undegraded":"2024-07-26T06:17:04.771350+0000","last_fullsized":"2024-07-26T06:17:04.771350+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:01:39.815405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300509+0000","last_change":"2024-07-26T06:17:03.305706+0000","last_active":"2024-07-26T06:17:04.300509+0000","last_peered":"2024-07-26T06:17:04.300509+0000","last_clean":"2024-07-26T06:17:04.300509+0000","last_became_active":"2024-07-26T06:17:03.305437+0000","last_became_peered":"2024-07-26T06:17:03.305437+0000","last_unstale":"2024-07-26T06:17:04.300509+0000","last_undegraded":"2024-07-26T06:17:04.300509+0000","last_fullsized":"2024-07-26T06:17:04.300509+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:56.646919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709270+0000","last_change":"2024-07-26T06:17:03.302600+0000","last_active":"2024-07-26T06:17:04.709270+0000","last_peered":"2024-07-26T06:17:04.709270+0000","last_clean":"2024-07-26T06:17:04.709270+0000","last_became_active":"2024-07-26T06:17:03.302275+0000","last_became_peered":"2024-07-26T06:17:03.302275+0000","last_unstale":"2024-07-26T06:17:04.709270+0000","last_undegraded":"2024-07-26T06:17:04.709270+0000","last_fullsized":"2024-07-26T06:17:04.709270+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:53:54.792889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.770701+0000","last_change":"2024-07-26T06:17:03.306048+0000","last_active":"2024-07-26T06:17:04.770701+0000","last_peered":"2024-07-26T06:17:04.770701+0000","last_clean":"2024-07-26T06:17:04.770701+0000","last_became_active":"2024-07-26T06:17:03.305759+0000","last_became_peered":"2024-07-26T06:17:03.305759+0000","last_unstale":"2024-07-26T06:17:04.770701+0000","last_undegraded":"2024-07-26T06:17:04.770701+0000","last_fullsized":"2024-07-26T06:17:04.770701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:24:54.429525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032892+0000","last_change":"2024-07-26T06:17:03.320576+0000","last_active":"2024-07-26T06:17:05.032892+0000","last_peered":"2024-07-26T06:17:05.032892+0000","last_clean":"2024-07-26T06:17:05.032892+0000","last_became_active":"2024-07-26T06:17:03.318866+0000","last_became_peered":"2024-07-26T06:17:03.318866+0000","last_unstale":"2024-07-26T06:17:05.032892+0000","last_undegraded":"2024-07-26T06:17:05.032892+0000","last_fullsized":"2024-07-26T06:17:05.032892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:09.981777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033981+0000","last_change":"2024-07-26T06:17:03.317763+0000","last_active":"2024-07-26T06:17:05.033981+0000","last_peered":"2024-07-26T06:17:05.033981+0000","last_clean":"2024-07-26T06:17:05.033981+0000","last_became_active":"2024-07-26T06:17:03.317382+0000","last_became_peered":"2024-07-26T06:17:03.317382+0000","last_unstale":"2024-07-26T06:17:05.033981+0000","last_undegraded":"2024-07-26T06:17:05.033981+0000","last_fullsized":"2024-07-26T06:17:05.033981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:25:43.511989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034337+0000","last_change":"2024-07-26T06:17:02.287249+0000","last_active":"2024-07-26T06:17:05.034337+0000","last_peered":"2024-07-26T06:17:05.034337+0000","last_clean":"2024-07-26T06:17:05.034337+0000","last_became_active":"2024-07-26T06:17:02.286944+0000","last_became_peered":"2024-07-26T06:17:02.286944+0000","last_unstale":"2024-07-26T06:17:05.034337+0000","last_undegraded":"2024-07-26T06:17:05.034337+0000","last_fullsized":"2024-07-26T06:17:05.034337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:42.996320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.298876+0000","last_change":"2024-07-26T06:17:03.309496+0000","last_active":"2024-07-26T06:17:04.298876+0000","last_peered":"2024-07-26T06:17:04.298876+0000","last_clean":"2024-07-26T06:17:04.298876+0000","last_became_active":"2024-07-26T06:17:03.308980+0000","last_became_peered":"2024-07-26T06:17:03.308980+0000","last_unstale":"2024-07-26T06:17:04.298876+0000","last_undegraded":"2024-07-26T06:17:04.298876+0000","last_fullsized":"2024-07-26T06:17:04.298876+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:07:43.865610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301295+0000","last_change":"2024-07-26T06:17:02.274447+0000","last_active":"2024-07-26T06:17:04.301295+0000","last_peered":"2024-07-26T06:17:04.301295+0000","last_clean":"2024-07-26T06:17:04.301295+0000","last_became_active":"2024-07-26T06:17:02.271770+0000","last_became_peered":"2024-07-26T06:17:02.271770+0000","last_unstale":"2024-07-26T06:17:04.301295+0000","last_undegraded":"2024-07-26T06:17:04.301295+0000","last_fullsized":"2024-07-26T06:17:04.301295+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:30:33.877813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033411+0000","last_change":"2024-07-26T06:17:03.322099+0000","last_active":"2024-07-26T06:17:05.033411+0000","last_peered":"2024-07-26T06:17:05.033411+0000","last_clean":"2024-07-26T06:17:05.033411+0000","last_became_active":"2024-07-26T06:17:03.321656+0000","last_became_peered":"2024-07-26T06:17:03.321656+0000","last_unstale":"2024-07-26T06:17:05.033411+0000","last_undegraded":"2024-07-26T06:17:05.033411+0000","last_fullsized":"2024-07-26T06:17:05.033411+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:53:54.792889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033770+0000","last_change":"2024-07-26T06:17:02.284029+0000","last_active":"2024-07-26T06:17:05.033770+0000","last_peered":"2024-07-26T06:17:05.033770+0000","last_clean":"2024-07-26T06:17:05.033770+0000","last_became_active":"2024-07-26T06:17:02.283729+0000","last_became_peered":"2024-07-26T06:17:02.283729+0000","last_unstale":"2024-07-26T06:17:05.033770+0000","last_undegraded":"2024-07-26T06:17:05.033770+0000","last_fullsized":"2024-07-26T06:17:05.033770+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:04:47.224395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033483+0000","last_change":"2024-07-26T06:17:03.320482+0000","last_active":"2024-07-26T06:17:05.033483+0000","last_peered":"2024-07-26T06:17:05.033483+0000","last_clean":"2024-07-26T06:17:05.033483+0000","last_became_active":"2024-07-26T06:17:03.318753+0000","last_became_peered":"2024-07-26T06:17:03.318753+0000","last_unstale":"2024-07-26T06:17:05.033483+0000","last_undegraded":"2024-07-26T06:17:05.033483+0000","last_fullsized":"2024-07-26T06:17:05.033483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:27.548776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035152+0000","last_change":"2024-07-26T06:17:02.283626+0000","last_active":"2024-07-26T06:17:05.035152+0000","last_peered":"2024-07-26T06:17:05.035152+0000","last_clean":"2024-07-26T06:17:05.035152+0000","last_became_active":"2024-07-26T06:17:02.283345+0000","last_became_peered":"2024-07-26T06:17:02.283345+0000","last_unstale":"2024-07-26T06:17:05.035152+0000","last_undegraded":"2024-07-26T06:17:05.035152+0000","last_fullsized":"2024-07-26T06:17:05.035152+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:18:07.513317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300243+0000","last_change":"2024-07-26T06:17:03.303310+0000","last_active":"2024-07-26T06:17:04.300243+0000","last_peered":"2024-07-26T06:17:04.300243+0000","last_clean":"2024-07-26T06:17:04.300243+0000","last_became_active":"2024-07-26T06:17:03.299794+0000","last_became_peered":"2024-07-26T06:17:03.299794+0000","last_unstale":"2024-07-26T06:17:04.300243+0000","last_undegraded":"2024-07-26T06:17:04.300243+0000","last_fullsized":"2024-07-26T06:17:04.300243+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:23.686053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300063+0000","last_change":"2024-07-26T06:17:02.286258+0000","last_active":"2024-07-26T06:17:04.300063+0000","last_peered":"2024-07-26T06:17:04.300063+0000","last_clean":"2024-07-26T06:17:04.300063+0000","last_became_active":"2024-07-26T06:17:02.285558+0000","last_became_peered":"2024-07-26T06:17:02.285558+0000","last_unstale":"2024-07-26T06:17:04.300063+0000","last_undegraded":"2024-07-26T06:17:04.300063+0000","last_fullsized":"2024-07-26T06:17:04.300063+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:43:32.476836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299503+0000","last_change":"2024-07-26T06:17:03.313226+0000","last_active":"2024-07-26T06:17:04.299503+0000","last_peered":"2024-07-26T06:17:04.299503+0000","last_clean":"2024-07-26T06:17:04.299503+0000","last_became_active":"2024-07-26T06:17:03.312960+0000","last_became_peered":"2024-07-26T06:17:03.312960+0000","last_unstale":"2024-07-26T06:17:04.299503+0000","last_undegraded":"2024-07-26T06:17:04.299503+0000","last_fullsized":"2024-07-26T06:17:04.299503+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:10.397906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033183+0000","last_change":"2024-07-26T06:17:02.282252+0000","last_active":"2024-07-26T06:17:05.033183+0000","last_peered":"2024-07-26T06:17:05.033183+0000","last_clean":"2024-07-26T06:17:05.033183+0000","last_became_active":"2024-07-26T06:17:02.281668+0000","last_became_peered":"2024-07-26T06:17:02.281668+0000","last_unstale":"2024-07-26T06:17:05.033183+0000","last_undegraded":"2024-07-26T06:17:05.033183+0000","last_fullsized":"2024-07-26T06:17:05.033183+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:02.128487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032957+0000","last_change":"2024-07-26T06:17:03.302996+0000","last_active":"2024-07-26T06:17:05.032957+0000","last_peered":"2024-07-26T06:17:05.032957+0000","last_clean":"2024-07-26T06:17:05.032957+0000","last_became_active":"2024-07-26T06:17:03.302488+0000","last_became_peered":"2024-07-26T06:17:03.302488+0000","last_unstale":"2024-07-26T06:17:05.032957+0000","last_undegraded":"2024-07-26T06:17:05.032957+0000","last_fullsized":"2024-07-26T06:17:05.032957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:40.829579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034233+0000","last_change":"2024-07-26T06:17:02.283642+0000","last_active":"2024-07-26T06:17:05.034233+0000","last_peered":"2024-07-26T06:17:05.034233+0000","last_clean":"2024-07-26T06:17:05.034233+0000","last_became_active":"2024-07-26T06:17:02.283221+0000","last_became_peered":"2024-07-26T06:17:02.283221+0000","last_unstale":"2024-07-26T06:17:05.034233+0000","last_undegraded":"2024-07-26T06:17:05.034233+0000","last_fullsized":"2024-07-26T06:17:05.034233+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:01:46.054473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299766+0000","last_change":"2024-07-26T06:17:03.298188+0000","last_active":"2024-07-26T06:17:04.299766+0000","last_peered":"2024-07-26T06:17:04.299766+0000","last_clean":"2024-07-26T06:17:04.299766+0000","last_became_active":"2024-07-26T06:17:03.297568+0000","last_became_peered":"2024-07-26T06:17:03.297568+0000","last_unstale":"2024-07-26T06:17:04.299766+0000","last_undegraded":"2024-07-26T06:17:04.299766+0000","last_fullsized":"2024-07-26T06:17:04.299766+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:34.546964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299563+0000","last_change":"2024-07-26T06:17:02.278335+0000","last_active":"2024-07-26T06:17:04.299563+0000","last_peered":"2024-07-26T06:17:04.299563+0000","last_clean":"2024-07-26T06:17:04.299563+0000","last_became_active":"2024-07-26T06:17:02.277970+0000","last_became_peered":"2024-07-26T06:17:02.277970+0000","last_unstale":"2024-07-26T06:17:04.299563+0000","last_undegraded":"2024-07-26T06:17:04.299563+0000","last_fullsized":"2024-07-26T06:17:04.299563+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:49.704193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300986+0000","last_change":"2024-07-26T06:17:03.306713+0000","last_active":"2024-07-26T06:17:04.300986+0000","last_peered":"2024-07-26T06:17:04.300986+0000","last_clean":"2024-07-26T06:17:04.300986+0000","last_became_active":"2024-07-26T06:17:03.306408+0000","last_became_peered":"2024-07-26T06:17:03.306408+0000","last_unstale":"2024-07-26T06:17:04.300986+0000","last_undegraded":"2024-07-26T06:17:04.300986+0000","last_fullsized":"2024-07-26T06:17:04.300986+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:12:00.712738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710439+0000","last_change":"2024-07-26T06:17:02.279660+0000","last_active":"2024-07-26T06:17:04.710439+0000","last_peered":"2024-07-26T06:17:04.710439+0000","last_clean":"2024-07-26T06:17:04.710439+0000","last_became_active":"2024-07-26T06:17:02.279264+0000","last_became_peered":"2024-07-26T06:17:02.279264+0000","last_unstale":"2024-07-26T06:17:04.710439+0000","last_undegraded":"2024-07-26T06:17:04.710439+0000","last_fullsized":"2024-07-26T06:17:04.710439+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:10.291783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710163+0000","last_change":"2024-07-26T06:17:03.307252+0000","last_active":"2024-07-26T06:17:04.710163+0000","last_peered":"2024-07-26T06:17:04.710163+0000","last_clean":"2024-07-26T06:17:04.710163+0000","last_became_active":"2024-07-26T06:17:03.306886+0000","last_became_peered":"2024-07-26T06:17:03.306886+0000","last_unstale":"2024-07-26T06:17:04.710163+0000","last_undegraded":"2024-07-26T06:17:04.710163+0000","last_fullsized":"2024-07-26T06:17:04.710163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:52:44.453577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301403+0000","last_change":"2024-07-26T06:17:02.261989+0000","last_active":"2024-07-26T06:17:04.301403+0000","last_peered":"2024-07-26T06:17:04.301403+0000","last_clean":"2024-07-26T06:17:04.301403+0000","last_became_active":"2024-07-26T06:17:02.261065+0000","last_became_peered":"2024-07-26T06:17:02.261065+0000","last_unstale":"2024-07-26T06:17:04.301403+0000","last_undegraded":"2024-07-26T06:17:04.301403+0000","last_fullsized":"2024-07-26T06:17:04.301403+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:43:47.202970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.298980+0000","last_change":"2024-07-26T06:17:03.308295+0000","last_active":"2024-07-26T06:17:04.298980+0000","last_peered":"2024-07-26T06:17:04.298980+0000","last_clean":"2024-07-26T06:17:04.298980+0000","last_became_active":"2024-07-26T06:17:03.307988+0000","last_became_peered":"2024-07-26T06:17:03.307988+0000","last_unstale":"2024-07-26T06:17:04.298980+0000","last_undegraded":"2024-07-26T06:17:04.298980+0000","last_fullsized":"2024-07-26T06:17:04.298980+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:30:28.411612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710210+0000","last_change":"2024-07-26T06:17:02.275540+0000","last_active":"2024-07-26T06:17:04.710210+0000","last_peered":"2024-07-26T06:17:04.710210+0000","last_clean":"2024-07-26T06:17:04.710210+0000","last_became_active":"2024-07-26T06:17:02.275124+0000","last_became_peered":"2024-07-26T06:17:02.275124+0000","last_unstale":"2024-07-26T06:17:04.710210+0000","last_undegraded":"2024-07-26T06:17:04.710210+0000","last_fullsized":"2024-07-26T06:17:04.710210+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:18:07.513317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709956+0000","last_change":"2024-07-26T06:17:03.307936+0000","last_active":"2024-07-26T06:17:04.709956+0000","last_peered":"2024-07-26T06:17:04.709956+0000","last_clean":"2024-07-26T06:17:04.709956+0000","last_became_active":"2024-07-26T06:17:03.307606+0000","last_became_peered":"2024-07-26T06:17:03.307606+0000","last_unstale":"2024-07-26T06:17:04.709956+0000","last_undegraded":"2024-07-26T06:17:04.709956+0000","last_fullsized":"2024-07-26T06:17:04.709956+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:56:52.744638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300366+0000","last_change":"2024-07-26T06:17:02.282762+0000","last_active":"2024-07-26T06:17:04.300366+0000","last_peered":"2024-07-26T06:17:04.300366+0000","last_clean":"2024-07-26T06:17:04.300366+0000","last_became_active":"2024-07-26T06:17:02.282445+0000","last_became_peered":"2024-07-26T06:17:02.282445+0000","last_unstale":"2024-07-26T06:17:04.300366+0000","last_undegraded":"2024-07-26T06:17:04.300366+0000","last_fullsized":"2024-07-26T06:17:04.300366+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:19:52.874876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034107+0000","last_change":"2024-07-26T06:17:03.312364+0000","last_active":"2024-07-26T06:17:05.034107+0000","last_peered":"2024-07-26T06:17:05.034107+0000","last_clean":"2024-07-26T06:17:05.034107+0000","last_became_active":"2024-07-26T06:17:03.312110+0000","last_became_peered":"2024-07-26T06:17:03.312110+0000","last_unstale":"2024-07-26T06:17:05.034107+0000","last_undegraded":"2024-07-26T06:17:05.034107+0000","last_fullsized":"2024-07-26T06:17:05.034107+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:09.981777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299557+0000","last_change":"2024-07-26T06:17:02.261481+0000","last_active":"2024-07-26T06:17:04.299557+0000","last_peered":"2024-07-26T06:17:04.299557+0000","last_clean":"2024-07-26T06:17:04.299557+0000","last_became_active":"2024-07-26T06:17:02.260963+0000","last_became_peered":"2024-07-26T06:17:02.260963+0000","last_unstale":"2024-07-26T06:17:04.299557+0000","last_undegraded":"2024-07-26T06:17:04.299557+0000","last_fullsized":"2024-07-26T06:17:04.299557+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:50:33.879393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709862+0000","last_change":"2024-07-26T06:17:03.302686+0000","last_active":"2024-07-26T06:17:04.709862+0000","last_peered":"2024-07-26T06:17:04.709862+0000","last_clean":"2024-07-26T06:17:04.709862+0000","last_became_active":"2024-07-26T06:17:03.302413+0000","last_became_peered":"2024-07-26T06:17:03.302413+0000","last_unstale":"2024-07-26T06:17:04.709862+0000","last_undegraded":"2024-07-26T06:17:04.709862+0000","last_fullsized":"2024-07-26T06:17:04.709862+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:46:33.423315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032821+0000","last_change":"2024-07-26T06:17:02.278586+0000","last_active":"2024-07-26T06:17:05.032821+0000","last_peered":"2024-07-26T06:17:05.032821+0000","last_clean":"2024-07-26T06:17:05.032821+0000","last_became_active":"2024-07-26T06:17:02.278327+0000","last_became_peered":"2024-07-26T06:17:02.278327+0000","last_unstale":"2024-07-26T06:17:05.032821+0000","last_undegraded":"2024-07-26T06:17:05.032821+0000","last_fullsized":"2024-07-26T06:17:05.032821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:10:43.414602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299775+0000","last_change":"2024-07-26T06:17:03.309411+0000","last_active":"2024-07-26T06:17:04.299775+0000","last_peered":"2024-07-26T06:17:04.299775+0000","last_clean":"2024-07-26T06:17:04.299775+0000","last_became_active":"2024-07-26T06:17:03.308603+0000","last_became_peered":"2024-07-26T06:17:03.308603+0000","last_unstale":"2024-07-26T06:17:04.299775+0000","last_undegraded":"2024-07-26T06:17:04.299775+0000","last_fullsized":"2024-07-26T06:17:04.299775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:35:46.209528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034060+0000","last_change":"2024-07-26T06:17:02.278462+0000","last_active":"2024-07-26T06:17:05.034060+0000","last_peered":"2024-07-26T06:17:05.034060+0000","last_clean":"2024-07-26T06:17:05.034060+0000","last_became_active":"2024-07-26T06:17:02.278115+0000","last_became_peered":"2024-07-26T06:17:02.278115+0000","last_unstale":"2024-07-26T06:17:05.034060+0000","last_undegraded":"2024-07-26T06:17:05.034060+0000","last_fullsized":"2024-07-26T06:17:05.034060+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:22:35.226528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710591+0000","last_change":"2024-07-26T06:17:03.308088+0000","last_active":"2024-07-26T06:17:04.710591+0000","last_peered":"2024-07-26T06:17:04.710591+0000","last_clean":"2024-07-26T06:17:04.710591+0000","last_became_active":"2024-07-26T06:17:03.303876+0000","last_became_peered":"2024-07-26T06:17:03.303876+0000","last_unstale":"2024-07-26T06:17:04.710591+0000","last_undegraded":"2024-07-26T06:17:04.710591+0000","last_fullsized":"2024-07-26T06:17:04.710591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:11:53.157038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":22,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709149+0000","last_change":"2024-07-26T06:17:02.285212+0000","last_active":"2024-07-26T06:17:04.709149+0000","last_peered":"2024-07-26T06:17:04.709149+0000","last_clean":"2024-07-26T06:17:04.709149+0000","last_became_active":"2024-07-26T06:17:02.284781+0000","last_became_peered":"2024-07-26T06:17:02.284781+0000","last_unstale":"2024-07-26T06:17:04.709149+0000","last_undegraded":"2024-07-26T06:17:04.709149+0000","last_fullsized":"2024-07-26T06:17:04.709149+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:02.128487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710085+0000","last_change":"2024-07-26T06:17:03.305771+0000","last_active":"2024-07-26T06:17:04.710085+0000","last_peered":"2024-07-26T06:17:04.710085+0000","last_clean":"2024-07-26T06:17:04.710085+0000","last_became_active":"2024-07-26T06:17:03.305505+0000","last_became_peered":"2024-07-26T06:17:03.305505+0000","last_unstale":"2024-07-26T06:17:04.710085+0000","last_undegraded":"2024-07-26T06:17:04.710085+0000","last_fullsized":"2024-07-26T06:17:04.710085+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:34:16.967573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301547+0000","last_change":"2024-07-26T06:17:02.261246+0000","last_active":"2024-07-26T06:17:04.301547+0000","last_peered":"2024-07-26T06:17:04.301547+0000","last_clean":"2024-07-26T06:17:04.301547+0000","last_became_active":"2024-07-26T06:17:02.260787+0000","last_became_peered":"2024-07-26T06:17:02.260787+0000","last_unstale":"2024-07-26T06:17:04.301547+0000","last_undegraded":"2024-07-26T06:17:04.301547+0000","last_fullsized":"2024-07-26T06:17:04.301547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:08.777869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035123+0000","last_change":"2024-07-26T06:17:03.313198+0000","last_active":"2024-07-26T06:17:05.035123+0000","last_peered":"2024-07-26T06:17:05.035123+0000","last_clean":"2024-07-26T06:17:05.035123+0000","last_became_active":"2024-07-26T06:17:03.312992+0000","last_became_peered":"2024-07-26T06:17:03.312992+0000","last_unstale":"2024-07-26T06:17:05.035123+0000","last_undegraded":"2024-07-26T06:17:05.035123+0000","last_fullsized":"2024-07-26T06:17:05.035123+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:53:54.792889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034063+0000","last_change":"2024-07-26T06:17:02.280600+0000","last_active":"2024-07-26T06:17:05.034063+0000","last_peered":"2024-07-26T06:17:05.034063+0000","last_clean":"2024-07-26T06:17:05.034063+0000","last_became_active":"2024-07-26T06:17:02.280312+0000","last_became_peered":"2024-07-26T06:17:02.280312+0000","last_unstale":"2024-07-26T06:17:05.034063+0000","last_undegraded":"2024-07-26T06:17:05.034063+0000","last_fullsized":"2024-07-26T06:17:05.034063+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:45.506404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300944+0000","last_change":"2024-07-26T06:17:03.301792+0000","last_active":"2024-07-26T06:17:04.300944+0000","last_peered":"2024-07-26T06:17:04.300944+0000","last_clean":"2024-07-26T06:17:04.300944+0000","last_became_active":"2024-07-26T06:17:03.301420+0000","last_became_peered":"2024-07-26T06:17:03.301420+0000","last_unstale":"2024-07-26T06:17:04.300944+0000","last_undegraded":"2024-07-26T06:17:04.300944+0000","last_fullsized":"2024-07-26T06:17:04.300944+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:40.822120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035052+0000","last_change":"2024-07-26T06:17:02.282104+0000","last_active":"2024-07-26T06:17:05.035052+0000","last_peered":"2024-07-26T06:17:05.035052+0000","last_clean":"2024-07-26T06:17:05.035052+0000","last_became_active":"2024-07-26T06:17:02.281805+0000","last_became_peered":"2024-07-26T06:17:02.281805+0000","last_unstale":"2024-07-26T06:17:05.035052+0000","last_undegraded":"2024-07-26T06:17:05.035052+0000","last_fullsized":"2024-07-26T06:17:05.035052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:46:32.712965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710226+0000","last_change":"2024-07-26T06:17:03.314179+0000","last_active":"2024-07-26T06:17:04.710226+0000","last_peered":"2024-07-26T06:17:04.710226+0000","last_clean":"2024-07-26T06:17:04.710226+0000","last_became_active":"2024-07-26T06:17:03.311141+0000","last_became_peered":"2024-07-26T06:17:03.311141+0000","last_unstale":"2024-07-26T06:17:04.710226+0000","last_undegraded":"2024-07-26T06:17:04.710226+0000","last_fullsized":"2024-07-26T06:17:04.710226+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:22:02.959859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033666+0000","last_change":"2024-07-26T06:17:02.279333+0000","last_active":"2024-07-26T06:17:05.033666+0000","last_peered":"2024-07-26T06:17:05.033666+0000","last_clean":"2024-07-26T06:17:05.033666+0000","last_became_active":"2024-07-26T06:17:02.279019+0000","last_became_peered":"2024-07-26T06:17:02.279019+0000","last_unstale":"2024-07-26T06:17:05.033666+0000","last_undegraded":"2024-07-26T06:17:05.033666+0000","last_fullsized":"2024-07-26T06:17:05.033666+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:46:32.712965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034134+0000","last_change":"2024-07-26T06:17:03.308759+0000","last_active":"2024-07-26T06:17:05.034134+0000","last_peered":"2024-07-26T06:17:05.034134+0000","last_clean":"2024-07-26T06:17:05.034134+0000","last_became_active":"2024-07-26T06:17:03.306805+0000","last_became_peered":"2024-07-26T06:17:03.306805+0000","last_unstale":"2024-07-26T06:17:05.034134+0000","last_undegraded":"2024-07-26T06:17:05.034134+0000","last_fullsized":"2024-07-26T06:17:05.034134+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:09.874814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300658+0000","last_change":"2024-07-26T06:17:02.279573+0000","last_active":"2024-07-26T06:17:04.300658+0000","last_peered":"2024-07-26T06:17:04.300658+0000","last_clean":"2024-07-26T06:17:04.300658+0000","last_became_active":"2024-07-26T06:17:02.278792+0000","last_became_peered":"2024-07-26T06:17:02.278792+0000","last_unstale":"2024-07-26T06:17:04.300658+0000","last_undegraded":"2024-07-26T06:17:04.300658+0000","last_fullsized":"2024-07-26T06:17:04.300658+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:54.322976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032951+0000","last_change":"2024-07-26T06:17:03.313546+0000","last_active":"2024-07-26T06:17:05.032951+0000","last_peered":"2024-07-26T06:17:05.032951+0000","last_clean":"2024-07-26T06:17:05.032951+0000","last_became_active":"2024-07-26T06:17:03.313201+0000","last_became_peered":"2024-07-26T06:17:03.313201+0000","last_unstale":"2024-07-26T06:17:05.032951+0000","last_undegraded":"2024-07-26T06:17:05.032951+0000","last_fullsized":"2024-07-26T06:17:05.032951+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:09.874814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.298345+0000","last_change":"2024-07-26T06:17:02.267372+0000","last_active":"2024-07-26T06:17:04.298345+0000","last_peered":"2024-07-26T06:17:04.298345+0000","last_clean":"2024-07-26T06:17:04.298345+0000","last_became_active":"2024-07-26T06:17:02.266449+0000","last_became_peered":"2024-07-26T06:17:02.266449+0000","last_unstale":"2024-07-26T06:17:04.298345+0000","last_undegraded":"2024-07-26T06:17:04.298345+0000","last_fullsized":"2024-07-26T06:17:04.298345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:54.513721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032834+0000","last_change":"2024-07-26T06:17:03.314246+0000","last_active":"2024-07-26T06:17:05.032834+0000","last_peered":"2024-07-26T06:17:05.032834+0000","last_clean":"2024-07-26T06:17:05.032834+0000","last_became_active":"2024-07-26T06:17:03.313967+0000","last_became_peered":"2024-07-26T06:17:03.313967+0000","last_unstale":"2024-07-26T06:17:05.032834+0000","last_undegraded":"2024-07-26T06:17:05.032834+0000","last_fullsized":"2024-07-26T06:17:05.032834+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:20:22.884388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033755+0000","last_change":"2024-07-26T06:17:02.284878+0000","last_active":"2024-07-26T06:17:05.033755+0000","last_peered":"2024-07-26T06:17:05.033755+0000","last_clean":"2024-07-26T06:17:05.033755+0000","last_became_active":"2024-07-26T06:17:02.284572+0000","last_became_peered":"2024-07-26T06:17:02.284572+0000","last_unstale":"2024-07-26T06:17:05.033755+0000","last_undegraded":"2024-07-26T06:17:05.033755+0000","last_fullsized":"2024-07-26T06:17:05.033755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:02.128487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035055+0000","last_change":"2024-07-26T06:17:03.299141+0000","last_active":"2024-07-26T06:17:05.035055+0000","last_peered":"2024-07-26T06:17:05.035055+0000","last_clean":"2024-07-26T06:17:05.035055+0000","last_became_active":"2024-07-26T06:17:03.298829+0000","last_became_peered":"2024-07-26T06:17:03.298829+0000","last_unstale":"2024-07-26T06:17:05.035055+0000","last_undegraded":"2024-07-26T06:17:05.035055+0000","last_fullsized":"2024-07-26T06:17:05.035055+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:21.198904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299087+0000","last_change":"2024-07-26T06:17:02.274040+0000","last_active":"2024-07-26T06:17:04.299087+0000","last_peered":"2024-07-26T06:17:04.299087+0000","last_clean":"2024-07-26T06:17:04.299087+0000","last_became_active":"2024-07-26T06:17:02.273593+0000","last_became_peered":"2024-07-26T06:17:02.273593+0000","last_unstale":"2024-07-26T06:17:04.299087+0000","last_undegraded":"2024-07-26T06:17:04.299087+0000","last_fullsized":"2024-07-26T06:17:04.299087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:18.532839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035271+0000","last_change":"2024-07-26T06:17:03.311818+0000","last_active":"2024-07-26T06:17:05.035271+0000","last_peered":"2024-07-26T06:17:05.035271+0000","last_clean":"2024-07-26T06:17:05.035271+0000","last_became_active":"2024-07-26T06:17:03.311543+0000","last_became_peered":"2024-07-26T06:17:03.311543+0000","last_unstale":"2024-07-26T06:17:05.035271+0000","last_undegraded":"2024-07-26T06:17:05.035271+0000","last_fullsized":"2024-07-26T06:17:05.035271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:59.791876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.034283+0000","last_change":"2024-07-26T06:17:02.286291+0000","last_active":"2024-07-26T06:17:05.034283+0000","last_peered":"2024-07-26T06:17:05.034283+0000","last_clean":"2024-07-26T06:17:05.034283+0000","last_became_active":"2024-07-26T06:17:02.285956+0000","last_became_peered":"2024-07-26T06:17:02.285956+0000","last_unstale":"2024-07-26T06:17:05.034283+0000","last_undegraded":"2024-07-26T06:17:05.034283+0000","last_fullsized":"2024-07-26T06:17:05.034283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:09.935590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.299075+0000","last_change":"2024-07-26T06:17:03.306919+0000","last_active":"2024-07-26T06:17:04.299075+0000","last_peered":"2024-07-26T06:17:04.299075+0000","last_clean":"2024-07-26T06:17:04.299075+0000","last_became_active":"2024-07-26T06:17:03.306664+0000","last_became_peered":"2024-07-26T06:17:03.306664+0000","last_unstale":"2024-07-26T06:17:04.299075+0000","last_undegraded":"2024-07-26T06:17:04.299075+0000","last_fullsized":"2024-07-26T06:17:04.299075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:50.708693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033309+0000","last_change":"2024-07-26T06:17:02.283880+0000","last_active":"2024-07-26T06:17:05.033309+0000","last_peered":"2024-07-26T06:17:05.033309+0000","last_clean":"2024-07-26T06:17:05.033309+0000","last_became_active":"2024-07-26T06:17:02.281865+0000","last_became_peered":"2024-07-26T06:17:02.281865+0000","last_unstale":"2024-07-26T06:17:05.033309+0000","last_undegraded":"2024-07-26T06:17:05.033309+0000","last_fullsized":"2024-07-26T06:17:05.033309+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:47:42.022939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.033917+0000","last_change":"2024-07-26T06:17:03.319002+0000","last_active":"2024-07-26T06:17:05.033917+0000","last_peered":"2024-07-26T06:17:05.033917+0000","last_clean":"2024-07-26T06:17:05.033917+0000","last_became_active":"2024-07-26T06:17:03.317280+0000","last_became_peered":"2024-07-26T06:17:03.317280+0000","last_unstale":"2024-07-26T06:17:05.033917+0000","last_undegraded":"2024-07-26T06:17:05.033917+0000","last_fullsized":"2024-07-26T06:17:05.033917+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:59.791876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.035412+0000","last_change":"2024-07-26T06:17:02.288315+0000","last_active":"2024-07-26T06:17:05.035412+0000","last_peered":"2024-07-26T06:17:05.035412+0000","last_clean":"2024-07-26T06:17:05.035412+0000","last_became_active":"2024-07-26T06:17:02.287947+0000","last_became_peered":"2024-07-26T06:17:02.287947+0000","last_unstale":"2024-07-26T06:17:05.035412+0000","last_undegraded":"2024-07-26T06:17:05.035412+0000","last_fullsized":"2024-07-26T06:17:05.035412+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:10.291783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300208+0000","last_change":"2024-07-26T06:17:03.311528+0000","last_active":"2024-07-26T06:17:04.300208+0000","last_peered":"2024-07-26T06:17:04.300208+0000","last_clean":"2024-07-26T06:17:04.300208+0000","last_became_active":"2024-07-26T06:17:03.310825+0000","last_became_peered":"2024-07-26T06:17:03.310825+0000","last_unstale":"2024-07-26T06:17:04.300208+0000","last_undegraded":"2024-07-26T06:17:04.300208+0000","last_fullsized":"2024-07-26T06:17:04.300208+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:01:45.282178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.298393+0000","last_change":"2024-07-26T06:17:02.284077+0000","last_active":"2024-07-26T06:17:04.298393+0000","last_peered":"2024-07-26T06:17:04.298393+0000","last_clean":"2024-07-26T06:17:04.298393+0000","last_became_active":"2024-07-26T06:17:02.282618+0000","last_became_peered":"2024-07-26T06:17:02.282618+0000","last_unstale":"2024-07-26T06:17:04.298393+0000","last_undegraded":"2024-07-26T06:17:04.298393+0000","last_fullsized":"2024-07-26T06:17:04.298393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:11.647869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.710305+0000","last_change":"2024-07-26T06:17:03.305807+0000","last_active":"2024-07-26T06:17:04.710305+0000","last_peered":"2024-07-26T06:17:04.710305+0000","last_clean":"2024-07-26T06:17:04.710305+0000","last_became_active":"2024-07-26T06:17:03.305528+0000","last_became_peered":"2024-07-26T06:17:03.305528+0000","last_unstale":"2024-07-26T06:17:04.710305+0000","last_undegraded":"2024-07-26T06:17:04.710305+0000","last_fullsized":"2024-07-26T06:17:04.710305+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:03:34.585075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.300362+0000","last_change":"2024-07-26T06:17:02.287640+0000","last_active":"2024-07-26T06:17:04.300362+0000","last_peered":"2024-07-26T06:17:04.300362+0000","last_clean":"2024-07-26T06:17:04.300362+0000","last_became_active":"2024-07-26T06:17:02.287326+0000","last_became_peered":"2024-07-26T06:17:02.287326+0000","last_unstale":"2024-07-26T06:17:04.300362+0000","last_undegraded":"2024-07-26T06:17:04.300362+0000","last_fullsized":"2024-07-26T06:17:04.300362+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:25:31.310402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.297894+0000","last_change":"2024-07-26T06:17:03.300743+0000","last_active":"2024-07-26T06:17:04.297894+0000","last_peered":"2024-07-26T06:17:04.297894+0000","last_clean":"2024-07-26T06:17:04.297894+0000","last_became_active":"2024-07-26T06:17:03.300422+0000","last_became_peered":"2024-07-26T06:17:03.300422+0000","last_unstale":"2024-07-26T06:17:04.297894+0000","last_undegraded":"2024-07-26T06:17:04.297894+0000","last_fullsized":"2024-07-26T06:17:04.297894+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:43:35.047107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.298377+0000","last_change":"2024-07-26T06:17:02.277502+0000","last_active":"2024-07-26T06:17:04.298377+0000","last_peered":"2024-07-26T06:17:04.298377+0000","last_clean":"2024-07-26T06:17:04.298377+0000","last_became_active":"2024-07-26T06:17:02.277175+0000","last_became_peered":"2024-07-26T06:17:02.277175+0000","last_unstale":"2024-07-26T06:17:04.298377+0000","last_undegraded":"2024-07-26T06:17:04.298377+0000","last_fullsized":"2024-07-26T06:17:04.298377+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:25.297774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301206+0000","last_change":"2024-07-26T06:17:03.297827+0000","last_active":"2024-07-26T06:17:04.301206+0000","last_peered":"2024-07-26T06:17:04.301206+0000","last_clean":"2024-07-26T06:17:04.301206+0000","last_became_active":"2024-07-26T06:17:03.297204+0000","last_became_peered":"2024-07-26T06:17:03.297204+0000","last_unstale":"2024-07-26T06:17:04.301206+0000","last_undegraded":"2024-07-26T06:17:04.301206+0000","last_fullsized":"2024-07-26T06:17:04.301206+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:35:05.069645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301400+0000","last_change":"2024-07-26T06:17:02.284953+0000","last_active":"2024-07-26T06:17:04.301400+0000","last_peered":"2024-07-26T06:17:04.301400+0000","last_clean":"2024-07-26T06:17:04.301400+0000","last_became_active":"2024-07-26T06:17:02.284659+0000","last_became_peered":"2024-07-26T06:17:02.284659+0000","last_unstale":"2024-07-26T06:17:04.301400+0000","last_undegraded":"2024-07-26T06:17:04.301400+0000","last_fullsized":"2024-07-26T06:17:04.301400+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:53:25.522161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.301471+0000","last_change":"2024-07-26T06:17:03.313788+0000","last_active":"2024-07-26T06:17:04.301471+0000","last_peered":"2024-07-26T06:17:04.301471+0000","last_clean":"2024-07-26T06:17:04.301471+0000","last_became_active":"2024-07-26T06:17:03.312806+0000","last_became_peered":"2024-07-26T06:17:03.312806+0000","last_unstale":"2024-07-26T06:17:04.301471+0000","last_undegraded":"2024-07-26T06:17:04.301471+0000","last_fullsized":"2024-07-26T06:17:04.301471+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:02.236608+0000","last_clean_scrub_stamp":"2024-07-26T06:17:02.236608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:04:13.469405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:05.032883+0000","last_change":"2024-07-26T06:17:02.285386+0000","last_active":"2024-07-26T06:17:05.032883+0000","last_peered":"2024-07-26T06:17:05.032883+0000","last_clean":"2024-07-26T06:17:05.032883+0000","last_became_active":"2024-07-26T06:17:02.285055+0000","last_became_peered":"2024-07-26T06:17:02.285055+0000","last_unstale":"2024-07-26T06:17:05.032883+0000","last_undegraded":"2024-07-26T06:17:05.032883+0000","last_fullsized":"2024-07-26T06:17:05.032883+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:01.232108+0000","last_clean_scrub_stamp":"2024-07-26T06:17:01.232108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:32:58.197161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":177,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-07-26T06:17:04.709201+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:17:04.709201+0000","last_peered":"2024-07-26T06:17:04.709201+0000","last_clean":"2024-07-26T06:17:04.709201+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:17:04.709201+0000","last_undegraded":"2024-07-26T06:17:04.709201+0000","last_fullsized":"2024-07-26T06:17:04.709201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509488,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343556,"statfs":{"total":96636764160,"available":96607801344,"internally_reserved":0,"allocated":1290240,"data_stored":876583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509489,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":688128,"data_stored":286055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":286707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509490,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":286111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1298432,"data_stored":878617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":700416,"data_stored":287667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542191,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":696320,"data_stored":286111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542193,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":286145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:17:08.253 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:17:08.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.512+0000 7f572a800640 1 -- 172.21.15.163:0/791184922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f572415d5a0 msgr2=0x7f57240ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.512+0000 7f572a800640 1 --2- 172.21.15.163:0/791184922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f572415d5a0 0x7f57240ce2f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5718009830 tx=0x7f571802a480 comp rx=0 tx=0).stop 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.512+0000 7f572a800640 1 -- 172.21.15.163:0/791184922 shutdown_connections 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.512+0000 7f572a800640 1 --2- 172.21.15.163:0/791184922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57240b1e80 0x7f57240b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.512+0000 7f572a800640 1 --2- 172.21.15.163:0/791184922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f572415d5a0 0x7f57240ce2f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.512+0000 7f572a800640 1 -- 172.21.15.163:0/791184922 >> 172.21.15.163:0/791184922 conn(0x7f57240cfda0 msgr2=0x7f57240d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.513+0000 7f572a800640 1 -- 172.21.15.163:0/791184922 shutdown_connections 2024-07-26T06:17:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.513+0000 7f572a800640 1 -- 172.21.15.163:0/791184922 wait complete. 2024-07-26T06:17:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.514+0000 7f572a800640 1 Processor -- start 2024-07-26T06:17:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.514+0000 7f572a800640 1 -- start start 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.515+0000 7f572a800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57240b1e80 0x7f57240ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.515+0000 7f572a800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57240ce830 0x7f57241f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.515+0000 7f572a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 0x7f57240c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.515+0000 7f572a800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57240b5130 con 0x7f572415d5a0 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.515+0000 7f572a800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f57240b4fb0 con 0x7f57240ce830 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.515+0000 7f572a800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f57240b52b0 con 0x7f57240b1e80 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 0x7f57240c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:08.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 0x7f57240c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50906/0 (socket says 172.21.15.163:50906) 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 -- 172.21.15.163:0/2465433444 learned_addr learned my addr 172.21.15.163:0/2465433444 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5728a00640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57240ce830 0x7f57241f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729400640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57240b1e80 0x7f57240ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 -- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57240b1e80 msgr2=0x7f57240ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57240b1e80 0x7f57240ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 -- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57240ce830 msgr2=0x7f57241f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57240ce830 0x7f57241f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.516+0000 7f5729e00640 1 -- 172.21.15.163:0/2465433444 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57241f5670 con 0x7f572415d5a0 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.517+0000 7f5729e00640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 0x7f57240c9270 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f571c00eb10 tx=0x7f571c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.517+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c004b20 con 0x7f572415d5a0 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.517+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f571c004ce0 con 0x7f572415d5a0 2024-07-26T06:17:08.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.517+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c005280 con 0x7f572415d5a0 2024-07-26T06:17:08.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.517+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57241f5950 con 0x7f572415d5a0 2024-07-26T06:17:08.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.517+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57241f5f10 con 0x7f572415d5a0 2024-07-26T06:17:08.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.518+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57240b57d0 con 0x7f572415d5a0 2024-07-26T06:17:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.519+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f571c005420 con 0x7f572415d5a0 2024-07-26T06:17:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.520+0000 7f5721600640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5704076480 0x7f5704078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.521+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f571c098890 con 0x7f572415d5a0 2024-07-26T06:17:08.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.521+0000 7f5729400640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5704076480 0x7f5704078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:08.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.522+0000 7f5729400640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5704076480 0x7f5704078940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5718034000 tx=0x7f5718004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:08.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.525+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f571c061e50 con 0x7f572415d5a0 2024-07-26T06:17:08.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.806+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f572415c7c0 con 0x7f572415d5a0 2024-07-26T06:17:08.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.812+0000 7f5721600640 1 -- 172.21.15.163:0/2465433444 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7f571c061750 con 0x7f572415d5a0 2024-07-26T06:17:08.813 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:17:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:17:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: caps mds = "allow" 2024-07-26T06:17:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: caps mon = "allow r" 2024-07-26T06:17:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-07-26T06:17:08.814 INFO:teuthology.orchestra.run.smithi163.stderr:updated caps for client.0 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5704076480 msgr2=0x7f5704078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5704076480 0x7f5704078940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5718034000 tx=0x7f5718004730 comp rx=0 tx=0).stop 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 msgr2=0x7f57240c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 0x7f57240c9270 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f571c00eb10 tx=0x7f571c00c2d0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 shutdown_connections 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5704076480 0x7f5704078940 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572415d5a0 0x7f57240c9270 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57240ce830 0x7f57241f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 --2- 172.21.15.163:0/2465433444 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57240b1e80 0x7f57240ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:08.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 >> 172.21.15.163:0/2465433444 conn(0x7f57240cfda0 msgr2=0x7f57240b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:08.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 shutdown_connections 2024-07-26T06:17:08.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:08.816+0000 7f572a800640 1 -- 172.21.15.163:0/2465433444 wait complete. 2024-07-26T06:17:08.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2398601462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 msgr2=0x7f8dd40b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2398601462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd40b6bd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc005fd0 tx=0x7f8dcc02a980 comp rx=0 tx=0).stop 2024-07-26T06:17:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2398601462 shutdown_connections 2024-07-26T06:17:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2398601462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd40b6bd0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2398601462 >> 172.21.15.163:0/2398601462 conn(0x7f8dd40cfda0 msgr2=0x7f8dd40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2398601462 shutdown_connections 2024-07-26T06:17:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.038+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2398601462 wait complete. 2024-07-26T06:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.040+0000 7f8dd8a00640 1 Processor -- start 2024-07-26T06:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.040+0000 7f8dd8a00640 1 -- start start 2024-07-26T06:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd8a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dd40b1e80 0x7f8dd414f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd8a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd41550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd8a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dd415d5a0 0x7f8dd40ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd8a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd40b5130 con 0x7f8dd40b23d0 2024-07-26T06:17:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd8a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd40b4fb0 con 0x7f8dd40b1e80 2024-07-26T06:17:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd8a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd40b52b0 con 0x7f8dd415d5a0 2024-07-26T06:17:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd41550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd41550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50922/0 (socket says 172.21.15.163:50922) 2024-07-26T06:17:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 -- 172.21.15.163:0/2625836697 learned_addr learned my addr 172.21.15.163:0/2625836697 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:09.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd3400640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dd40b1e80 0x7f8dd414f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd3e00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dd415d5a0 0x7f8dd40ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 -- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dd415d5a0 msgr2=0x7f8dd40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dd415d5a0 0x7f8dd40ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 -- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dd40b1e80 msgr2=0x7f8dd414f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dd40b1e80 0x7f8dd414f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dcbe00640 1 -- 172.21.15.163:0/2625836697 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd414fc50 con 0x7f8dd40b23d0 2024-07-26T06:17:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.041+0000 7f8dd3400640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dd40b1e80 0x7f8dd414f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.042+0000 7f8dcbe00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd41550a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00c430 tx=0x7f8dbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.042+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc011790 con 0x7f8dd40b23d0 2024-07-26T06:17:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.042+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dbc017070 con 0x7f8dd40b23d0 2024-07-26T06:17:09.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.042+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd41558f0 con 0x7f8dd40b23d0 2024-07-26T06:17:09.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.042+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc011d00 con 0x7f8dd40b23d0 2024-07-26T06:17:09.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.042+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd4155eb0 con 0x7f8dd40b23d0 2024-07-26T06:17:09.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.044+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd40b57b0 con 0x7f8dd40b23d0 2024-07-26T06:17:09.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.045+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f8dbc01c460 con 0x7f8dd40b23d0 2024-07-26T06:17:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.046+0000 7f8dcb400640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8db00764b0 0x7f8db0078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.046+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f8dbc097960 con 0x7f8dd40b23d0 2024-07-26T06:17:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.047+0000 7f8dd3400640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8db00764b0 0x7f8db0078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.048+0000 7f8dd3400640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8db00764b0 0x7f8db0078970 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40016a0 tx=0x7f8dc4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.051+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8dbc0610d0 con 0x7f8dd40b23d0 2024-07-26T06:17:09.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.278+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dd415b740 con 0x7f8dd40b23d0 2024-07-26T06:17:09.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.279+0000 7f8dcb400640 1 -- 172.21.15.163:0/2625836697 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f8dbc0690c0 con 0x7f8dd40b23d0 2024-07-26T06:17:09.280 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:09.281 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":35,"btime":"2024-07-26T06:17:06:319401+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:02.717555+0000","modified":"2024-07-26T06:17:05.791542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/3071317413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3071317413},{"type":"v1","addr":"172.21.15.164:6835","nonce":3071317413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5053,"qdb_cluster":[5053]},"id":4}]} 2024-07-26T06:17:09.281 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 35 2024-07-26T06:17:09.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8db00764b0 msgr2=0x7f8db0078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8db00764b0 0x7f8db0078970 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40016a0 tx=0x7f8dc4009040 comp rx=0 tx=0).stop 2024-07-26T06:17:09.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 msgr2=0x7f8dd41550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd41550a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00c430 tx=0x7f8dbc00c8f0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 shutdown_connections 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8db00764b0 0x7f8db0078970 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dd415d5a0 0x7f8dd40ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8dd40b23d0 0x7f8dd41550a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 --2- 172.21.15.163:0/2625836697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dd40b1e80 0x7f8dd414f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 >> 172.21.15.163:0/2625836697 conn(0x7f8dd40cfda0 msgr2=0x7f8dd40b8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 shutdown_connections 2024-07-26T06:17:09.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.282+0000 7f8dd8a00640 1 -- 172.21.15.163:0/2625836697 wait complete. 2024-07-26T06:17:09.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:17:02.717555+0000', 'modified': '2024-07-26T06:17:05.791542+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5053}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5053': {'gid': 5053, 'name': 'd', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6835/3071317413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 3071317413}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 3071317413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5053, 'qdb_cluster': [5053]} 2024-07-26T06:17:09.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:17:09.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-07-26T06:17:09.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 -- 172.21.15.163:0/1348986390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f64071c40 msgr2=0x7f9f640b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/1348986390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f64071c40 0x7f9f640b3710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c005fd0 tx=0x7f9f5c02a8b0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 -- 172.21.15.163:0/1348986390 shutdown_connections 2024-07-26T06:17:09.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/1348986390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f64071c40 0x7f9f640b3710 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/1348986390 >> v2:172.21.15.164:3300/0 conn(0x7f9f641556f0 0x7f9f640cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 -- 172.21.15.163:0/1348986390 >> 172.21.15.163:0/1348986390 conn(0x7f9f640b0e30 msgr2=0x7f9f640b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 -- 172.21.15.163:0/1348986390 shutdown_connections 2024-07-26T06:17:09.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.922+0000 7f9f6aa00640 1 -- 172.21.15.163:0/1348986390 wait complete. 2024-07-26T06:17:09.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.924+0000 7f9f6aa00640 1 Processor -- start 2024-07-26T06:17:09.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.924+0000 7f9f6aa00640 1 -- start start 2024-07-26T06:17:09.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f6aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 0x7f9f64158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f6aa00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f640ce2d0 0x7f9f641590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f6aa00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f641556f0 0x7f9f641e7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f6aa00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640b3fe0 con 0x7f9f64071c40 2024-07-26T06:17:09.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f6aa00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640b3e60 con 0x7f9f641556f0 2024-07-26T06:17:09.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f6aa00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9f640b4160 con 0x7f9f640ce2d0 2024-07-26T06:17:09.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f69600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 0x7f9f64158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.925+0000 7f9f69600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 0x7f9f64158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50924/0 (socket says 172.21.15.163:50924) 2024-07-26T06:17:09.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 -- 172.21.15.163:0/733267109 learned_addr learned my addr 172.21.15.163:0/733267109 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 -- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f640ce2d0 msgr2=0x7f9f641590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f640ce2d0 0x7f9f641590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 -- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f641556f0 msgr2=0x7f9f641e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f641556f0 0x7f9f641e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 -- 172.21.15.163:0/733267109 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f641e8000 con 0x7f9f64071c40 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f69600640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 0x7f9f64158b90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800e790 tx=0x7f9f5800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f61600640 1 -- 172.21.15.163:0/733267109 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5800eea0 con 0x7f9f64071c40 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f61600640 1 -- 172.21.15.163:0/733267109 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f5800cd50 con 0x7f9f64071c40 2024-07-26T06:17:09.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.926+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6413da30 con 0x7f9f64071c40 2024-07-26T06:17:09.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.927+0000 7f9f61600640 1 -- 172.21.15.163:0/733267109 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58010690 con 0x7f9f64071c40 2024-07-26T06:17:09.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.927+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6413dff0 con 0x7f9f64071c40 2024-07-26T06:17:09.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.929+0000 7f9f61600640 1 -- 172.21.15.163:0/733267109 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f9f58010830 con 0x7f9f64071c40 2024-07-26T06:17:09.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.930+0000 7f9f61600640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9f44076480 0x7f9f44078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.930+0000 7f9f61600640 1 -- 172.21.15.163:0/733267109 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9f580984b0 con 0x7f9f64071c40 2024-07-26T06:17:09.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.930+0000 7f9f6f012740 1 Processor -- start 2024-07-26T06:17:09.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.930+0000 7f9f6f012740 1 -- start start 2024-07-26T06:17:09.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f68c00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9f44076480 0x7f9f44078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad5690f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5692d90 0x559ad56932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad564de40 0x559ad5693820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f6f012740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559ad55633f0 con 0x559ad56735c0 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x559ad55be5c0 con 0x559ad564de40 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.931+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x559ad55adb60 con 0x559ad5692d90 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad5690f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad5690f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50930/0 (socket says 172.21.15.163:50930) 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 -- 172.21.15.163:0/3719513929 learned_addr learned my addr 172.21.15.163:0/3719513929 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f68c00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9f44076480 0x7f9f44078940 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9f6415c270 tx=0x7f9f50007a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f68c00640 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5692d90 0x559ad56932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 -- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5692d90 msgr2=0x559ad56932e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5692d90 0x559ad56932e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad564de40 0x559ad5693820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 -- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad564de40 msgr2=0x559ad5693820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad564de40 0x559ad5693820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 -- 172.21.15.163:0/3719513929 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ad5312850 con 0x559ad56735c0 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.932+0000 7f9f69600640 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad5690f80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9f58061070 tx=0x7f9f58061a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f49ad157eeb1c1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f55c00640 1 -- 172.21.15.163:0/3719513929 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58096aa0 con 0x559ad56735c0 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f55c00640 1 -- 172.21.15.163:0/3719513929 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f5806a140 con 0x559ad56735c0 2024-07-26T06:17:09.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f55c00640 1 -- 172.21.15.163:0/3719513929 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58069ef0 con 0x559ad56735c0 2024-07-26T06:17:09.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 -- 172.21.15.163:0/3719513929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 msgr2=0x559ad5690f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad5690f80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9f58061070 tx=0x7f9f58061a70 comp rx=0 tx=0).stop 2024-07-26T06:17:09.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 -- 172.21.15.163:0/3719513929 shutdown_connections 2024-07-26T06:17:09.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad564de40 0x559ad5693820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5692d90 0x559ad56932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3719513929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad5690f80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 -- 172.21.15.163:0/3719513929 >> 172.21.15.163:0/3719513929 conn(0x559ad56491a0 msgr2=0x559ad56915d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.933+0000 7f9f6f012740 1 -- 172.21.15.163:0/3719513929 shutdown_connections 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.934+0000 7f9f6f012740 1 -- 172.21.15.163:0/3719513929 wait complete. 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.934+0000 7f9f6f012740 1 -- start start 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.934+0000 7f9f6f012740 1 Processor -- start 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.935+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad567a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.935+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5646880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.935+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 0x559ad568cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.935+0000 7f9f6f012740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559ad5556210 con 0x559ad5646dc0 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x559ad55b1ba0 con 0x559ad56735c0 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x559ad5692720 con 0x559ad56a6370 2024-07-26T06:17:09.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6f012740 10 client.0 _ll_register_callbacks cb 0x559ad55568a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 0x559ad568cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 0x559ad568cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50934/0 (socket says 172.21.15.163:50934) 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 -- 172.21.15.163:0/3620907338 learned_addr learned my addr 172.21.15.163:0/3620907338 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f69600640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad567a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f68c00640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5646880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 -- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 msgr2=0x559ad5646880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5646880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 -- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 msgr2=0x559ad567a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad567a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f6a000640 1 -- 172.21.15.163:0/3620907338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ad53896b0 con 0x559ad5646dc0 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.936+0000 7f9f68c00640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5646880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 0x559ad568cc50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009cd0 tx=0x7f9f5c003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508b20d57a06176e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f4a000640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f4a000640 1 -- 172.21.15.163:0/3620907338 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c040070 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559ad55c0510 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f4a000640 1 -- 172.21.15.163:0/3620907338 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9f5c004450 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f4a000640 1 -- 172.21.15.163:0/3620907338 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9f5c04d050 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f6f012740 10 client.5835 fetch_fsmap learned FSMap version 35 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x559ad564a810 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.937+0000 7f9f4a000640 1 -- 172.21.15.163:0/3620907338 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x559ad55c0510 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f4a000640 1 -- 172.21.15.163:0/3620907338 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7f9f5c050020 con 0x559ad5646dc0 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f6f012740 10 client.5835 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T06:17:09.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f6f012740 10 client.5835 resolve_mds: resolved name 'd' to daemon mds.d 2024-07-26T06:17:09.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f6f012740 20 client.5835 populate_metadata read hostname 'smithi163' 2024-07-26T06:17:09.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad56b21e0 0x559ad56b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f6f012740 4 client.5835 mds_command: new command op to 5053 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:17:09.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 --> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ad55bbfb0 con 0x559ad56b21e0 2024-07-26T06:17:09.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.938+0000 7f9f69600640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad56b21e0 0x559ad56b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.939+0000 7f9f69600640 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad56b21e0 0x559ad56b45d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eefd4679b04a52e6 server_cookie=c1bdc0e69a930d59 in_seq=0 out_seq=0 2024-07-26T06:17:09.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.939+0000 7f9f4a000640 10 client.5835 ms_handle_connect on v2:172.21.15.164:6834/3071317413 2024-07-26T06:17:09.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.941+0000 7f9f4a000640 1 -- 172.21.15.163:0/3620907338 <== mds.0 v2:172.21.15.164:6834/3071317413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559ad55bbfb0 con 0x559ad56b21e0 2024-07-26T06:17:09.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.941+0000 7f9f4a000640 10 client.5835 handle_command_reply: tid=0 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 client.5835 shutdown 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 msgr2=0x559ad568cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 0x559ad568cc50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009cd0 tx=0x7f9f5c003f00 comp rx=0 tx=0).stop 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 shutdown_connections 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad56b21e0 0x559ad56b45d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad5646dc0 0x559ad568cc50 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5646880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3620907338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad567a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 >> 172.21.15.163:0/3620907338 conn(0x559ad564de40 msgr2=0x559ad5693220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f4a000640 0 client.5835 ms_handle_reset on v2:172.21.15.164:6834/3071317413 2024-07-26T06:17:09.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 shutdown_connections 2024-07-26T06:17:09.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 1 -- 172.21.15.163:0/3620907338 wait complete. 2024-07-26T06:17:09.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.942+0000 7f9f6f012740 20 client.5835 trim_cache size 0 max 16384 2024-07-26T06:17:09.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 Processor -- start 2024-07-26T06:17:09.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 -- start start 2024-07-26T06:17:09.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5646dc0 0x559ad5678fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad56794e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 0x559ad5647340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640cd9b0 con 0x559ad56a6370 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640ccab0 con 0x559ad56735c0 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.963+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9f640cbe10 con 0x559ad5646dc0 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 0x559ad5647340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 0x559ad5647340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50950/0 (socket says 172.21.15.163:50950) 2024-07-26T06:17:09.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 -- 172.21.15.163:0/3603831747 learned_addr learned my addr 172.21.15.163:0/3603831747 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f68c00640 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad56794e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f69600640 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5646dc0 0x559ad5678fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 -- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5646dc0 msgr2=0x559ad5678fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5646dc0 0x559ad5678fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 -- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 msgr2=0x559ad56794e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad56794e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f6a000640 1 -- 172.21.15.163:0/3603831747 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ad564add0 con 0x559ad56a6370 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.964+0000 7f9f68c00640 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad56794e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.965+0000 7f9f6a000640 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 0x559ad5647340 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c0045f0 tx=0x7f9f5c004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f58eab8efd1dc77 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.965+0000 7f9f55200640 1 -- 172.21.15.163:0/3603831747 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c040040 con 0x559ad56a6370 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.965+0000 7f9f55200640 1 -- 172.21.15.163:0/3603831747 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f5c004d40 con 0x559ad56a6370 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.965+0000 7f9f55200640 1 -- 172.21.15.163:0/3603831747 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c045710 con 0x559ad56a6370 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 -- 172.21.15.163:0/3603831747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 msgr2=0x559ad5647340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 0x559ad5647340 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c0045f0 tx=0x7f9f5c004620 comp rx=0 tx=0).stop 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 -- 172.21.15.163:0/3603831747 shutdown_connections 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56a6370 0x559ad5647340 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad56735c0 0x559ad56794e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 --2- 172.21.15.163:0/3603831747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad5646dc0 0x559ad5678fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 -- 172.21.15.163:0/3603831747 >> 172.21.15.163:0/3603831747 conn(0x559ad564de40 msgr2=0x559ad5642870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 -- 172.21.15.163:0/3603831747 shutdown_connections 2024-07-26T06:17:09.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.966+0000 7f9f6f012740 1 -- 172.21.15.163:0/3603831747 wait complete. 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.967+0000 7f9f6f012740 1 -- start start 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.967+0000 7f9f6f012740 1 Processor -- start 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad5646dc0 0x559ad567a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad568acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5713000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559ad564cff0 con 0x559ad56735c0 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640cc460 con 0x559ad5646dc0 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x559ad564b5d0 con 0x559ad56a6370 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f6f012740 10 client.0 _ll_register_callbacks cb 0x7f9f5c003da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:17:09.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f68c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad568acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f68c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad568acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50958/0 (socket says 172.21.15.163:50958) 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.968+0000 7f9f68c00640 1 -- 172.21.15.163:0/2052893445 learned_addr learned my addr 172.21.15.163:0/2052893445 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f69600640 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad5646dc0 0x559ad567a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f68c00640 1 -- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 msgr2=0x559ad5713000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f68c00640 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5713000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f68c00640 1 -- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad5646dc0 msgr2=0x559ad567a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f68c00640 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad5646dc0 0x559ad567a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f68c00640 1 -- 172.21.15.163:0/2052893445 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ad5691770 con 0x559ad56735c0 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f68c00640 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad568acb0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f50001e40 tx=0x7f9f50004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf25972f7d715ee server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f4a000640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f4a000640 1 -- 172.21.15.163:0/2052893445 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5001f5f0 con 0x559ad56735c0 2024-07-26T06:17:09.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f4a000640 1 -- 172.21.15.163:0/2052893445 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9f5001f790 con 0x559ad56735c0 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f5c004d40 con 0x559ad56735c0 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f4a000640 1 -- 172.21.15.163:0/2052893445 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9f5002d050 con 0x559ad56735c0 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.969+0000 7f9f6f012740 10 client.5841 fetch_fsmap learned FSMap version 35 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x559ad568b420 con 0x559ad56735c0 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f4a000640 1 -- 172.21.15.163:0/2052893445 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f5c004d40 con 0x559ad56735c0 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f4a000640 1 -- 172.21.15.163:0/2052893445 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7f9f50030020 con 0x559ad56735c0 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 10 client.5841 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 10 client.5841 resolve_mds: resolved name 'd' to daemon mds.d 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 20 client.5841 populate_metadata read hostname 'smithi163' 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad5687a40 0x559ad56759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:09.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 4 client.5841 mds_command: new command op to 5053 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:17:09.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.970+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 --> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] -- command(tid 0: {"prefix": "status"}) -- 0x559ad55b96f0 con 0x559ad5687a40 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.971+0000 7f9f69600640 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad5687a40 0x559ad56759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f69600640 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad5687a40 0x559ad56759b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96bfb565ad6b5e4 server_cookie=298fdef9a2b0ce8d in_seq=0 out_seq=0 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f4a000640 10 client.5841 ms_handle_connect on v2:172.21.15.164:6834/3071317413 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f4a000640 1 -- 172.21.15.163:0/2052893445 <== mds.0 v2:172.21.15.164:6834/3071317413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x559ad55b96f0 con 0x559ad5687a40 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f4a000640 10 client.5841 handle_command_reply: tid=0 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 client.5841 shutdown 2024-07-26T06:17:09.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 msgr2=0x559ad568acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad568acb0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f50001e40 tx=0x7f9f50004620 comp rx=0 tx=0).stop 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 shutdown_connections 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:6834/3071317413,v1:172.21.15.164:6835/3071317413] conn(0x559ad5687a40 0x559ad56759b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559ad56a6370 0x559ad5713000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559ad56735c0 0x559ad568acb0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 --2- 172.21.15.163:0/2052893445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559ad5646dc0 0x559ad567a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f4a000640 0 client.5841 ms_handle_reset on v2:172.21.15.164:6834/3071317413 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.972+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 >> 172.21.15.163:0/2052893445 conn(0x559ad564de40 msgr2=0x559ad5647340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 shutdown_connections 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6f012740 1 -- 172.21.15.163:0/2052893445 wait complete. 2024-07-26T06:17:09.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6f012740 20 client.5841 trim_cache size 0 max 16384 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 5053, 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 6.6997044990000001, 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 35, 2024-07-26T06:17:09.977 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 58, 2024-07-26T06:17:09.978 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 58, 2024-07-26T06:17:09.978 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 11.667227959 2024-07-26T06:17:09.978 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9f44076480 msgr2=0x7f9f44078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9f44076480 0x7f9f44078940 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9f6415c270 tx=0x7f9f50007a30 comp rx=0 tx=0).stop 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 msgr2=0x7f9f64158b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 0x7f9f64158b90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800e790 tx=0x7f9f5800ec50 comp rx=0 tx=0).stop 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 shutdown_connections 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9f44076480 0x7f9f44078940 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f641556f0 0x7f9f641e7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f640ce2d0 0x7f9f641590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 --2- 172.21.15.163:0/733267109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f64071c40 0x7f9f64158b90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.973+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 >> 172.21.15.163:0/733267109 conn(0x7f9f640b0e30 msgr2=0x7f9f6415df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:09.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.974+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 shutdown_connections 2024-07-26T06:17:09.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:09.974+0000 7f9f6aa00640 1 -- 172.21.15.163:0/733267109 wait complete. 2024-07-26T06:17:09.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:17:10.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39cd200640 1 -- 172.21.15.163:0/3860596659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 msgr2=0x7f39ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:10.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39cd200640 1 --2- 172.21.15.163:0/3860596659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39ac0051c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009e20 tx=0x7f39ac005830 comp rx=0 tx=0).stop 2024-07-26T06:17:10.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39c6800640 1 -- 172.21.15.163:0/3860596659 reap_dead start 2024-07-26T06:17:10.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39cd200640 1 -- 172.21.15.163:0/3860596659 shutdown_connections 2024-07-26T06:17:10.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39cd200640 1 -- 172.21.15.163:0/3860596659 >> 172.21.15.163:0/3860596659 conn(0x7f39c806d420 msgr2=0x7f39c806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:10.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39cd200640 1 -- 172.21.15.163:0/3860596659 shutdown_connections 2024-07-26T06:17:10.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.189+0000 7f39cd200640 1 -- 172.21.15.163:0/3860596659 wait complete. 2024-07-26T06:17:10.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.191+0000 7f39cd200640 1 Processor -- start 2024-07-26T06:17:10.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.191+0000 7f39cd200640 1 -- start start 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39cd200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39c8154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39cd200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39c8153bf0 0x7f39c81500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39cd200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39c8156580 0x7f39c8156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39cd200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f39c815ed20 con 0x7f39ac004d40 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39cd200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f39c815eea0 con 0x7f39c8153bf0 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39cd200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f39c815f020 con 0x7f39c8156580 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39c5e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39c8154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39c5e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39c8154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50968/0 (socket says 172.21.15.163:50968) 2024-07-26T06:17:10.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39c5e00640 1 -- 172.21.15.163:0/4118102029 learned_addr learned my addr 172.21.15.163:0/4118102029 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39c5e00640 1 -- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39c8156580 msgr2=0x7f39c8156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.192+0000 7f39c5e00640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39c8156580 0x7f39c8156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39c5e00640 1 -- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39c8153bf0 msgr2=0x7f39c81500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39c5e00640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39c8153bf0 0x7f39c81500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39c5e00640 1 -- 172.21.15.163:0/4118102029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c8150630 con 0x7f39ac004d40 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39c5e00640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39c8154dd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f39b800e740 tx=0x7f39b800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b800cb80 con 0x7f39ac004d40 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39b800ce10 con 0x7f39ac004d40 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8005280 con 0x7f39ac004d40 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c81554d0 con 0x7f39ac004d40 2024-07-26T06:17:10.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.193+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c8155a90 con 0x7f39ac004d40 2024-07-26T06:17:10.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.194+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c80b44f0 con 0x7f39ac004d40 2024-07-26T06:17:10.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.195+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f39b8005420 con 0x7f39ac004d40 2024-07-26T06:17:10.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.196+0000 7f39be000640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f39a80764b0 0x7f39a8078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:10.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.197+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f39b80981e0 con 0x7f39ac004d40 2024-07-26T06:17:10.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.197+0000 7f39c5400640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f39a80764b0 0x7f39a8078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:10.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.198+0000 7f39c5400640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f39a80764b0 0x7f39a8078970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39b4001860 tx=0x7f39b4001cd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:10.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.201+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f39b8016030 con 0x7f39ac004d40 2024-07-26T06:17:10.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.456+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f39c8069050 con 0x7f39ac004d40 2024-07-26T06:17:10.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.457+0000 7f39be000640 1 -- 172.21.15.163:0/4118102029 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f39b80617b0 con 0x7f39ac004d40 2024-07-26T06:17:10.458 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:17:10.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f39a80764b0 msgr2=0x7f39a8078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:10.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f39a80764b0 0x7f39a8078970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39b4001860 tx=0x7f39b4001cd0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 msgr2=0x7f39c8154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:10.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39c8154dd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f39b800e740 tx=0x7f39b800ec00 comp rx=0 tx=0).stop 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 shutdown_connections 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f39a80764b0 0x7f39a8078970 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39c8156580 0x7f39c8156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39c8153bf0 0x7f39c81500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 --2- 172.21.15.163:0/4118102029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39ac004d40 0x7f39c8154dd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 >> 172.21.15.163:0/4118102029 conn(0x7f39c806d420 msgr2=0x7f39c80b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.460+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 shutdown_connections 2024-07-26T06:17:10.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10.461+0000 7f39cd200640 1 -- 172.21.15.163:0/4118102029 wait complete. 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:17:10.482 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:17:10.483 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:17:10.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:10.483 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:17:10.533 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:17:10.533 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:10.534 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42374sec preferred_lft 42374sec 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:10.535 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:17:10.536 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:17:10.536 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.536 INFO:teuthology.orchestra.run.smithi163.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:10.536 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether c2:4f:6d:14:5d:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:17:10.536 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::c04f:6dff:fe14:5d96/64 scope link 2024-07-26T06:17:10.536 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:10.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:10.537 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:17:10.614 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:17:10.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:10.618 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:17:10.672 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:17:10.672 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:17:10.672 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:17:10.672 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:17:10.672 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:17:10.673 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:17:10.673 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:17:10.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:17:10.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:17:10.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:17:10.673 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:17:10.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:17:10.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:10.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:10.880 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:17:10.926 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:17:10.926 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:17:10.927 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:17:10.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:10.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:17:11.008 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:11.008 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:11.061 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:17:11.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:11.062 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:11.112 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:17:11.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:11.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:11.240 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:11.240 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:17:11.241 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:17:11.262 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:11.262 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:17:11.262 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:17:11.262 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:11.262 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:11.291 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:11.291 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:17:11.291 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:17:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:17:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:17:11.292 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:17:11.294 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:17:11.294 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:17:11.295 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:17:11.295 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:17:11.295 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:17:11.299 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:17:11.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:11.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:11.375 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:17:11.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:17:11.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:11.509 INFO:teuthology.orchestra.run.smithi163.stdout:client5119 2024-07-26T06:17:11.510 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:11.510 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:11.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f191a400640 1 -- 172.21.15.163:0/576793122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18fc003a80 msgr2=0x7f18fc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f191a400640 1 --2- 172.21.15.163:0/576793122 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18fc003a80 0x7f18fc003e80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f190800c6f0 tx=0x7f18fc005830 comp rx=0 tx=0).stop 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f1919a00640 1 -- 172.21.15.163:0/576793122 reap_dead start 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f191a400640 1 -- 172.21.15.163:0/576793122 shutdown_connections 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f191a400640 1 -- 172.21.15.163:0/576793122 >> 172.21.15.163:0/576793122 conn(0x7f19140b0e30 msgr2=0x7f19140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f191a400640 1 -- 172.21.15.163:0/576793122 shutdown_connections 2024-07-26T06:17:11.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.767+0000 7f191a400640 1 -- 172.21.15.163:0/576793122 wait complete. 2024-07-26T06:17:11.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.769+0000 7f191a400640 1 Processor -- start 2024-07-26T06:17:11.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.769+0000 7f191a400640 1 -- start start 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f191a400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18fc003a80 0x7f19141485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f191a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 0x7f19140cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f191a400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1914162fa0 0x7f1914163420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f191a400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1914164310 con 0x7f19140cd010 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f191a400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1914164190 con 0x7f1914162fa0 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f191a400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1914164490 con 0x7f18fc003a80 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f1913e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 0x7f19140cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:11.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f1913e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 0x7f19140cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50972/0 (socket says 172.21.15.163:50972) 2024-07-26T06:17:11.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f1913e00640 1 -- 172.21.15.163:0/4269715487 learned_addr learned my addr 172.21.15.163:0/4269715487 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:11.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f1919000640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18fc003a80 0x7f19141485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:11.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.770+0000 7f1919a00640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1914162fa0 0x7f1914163420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1913e00640 1 -- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18fc003a80 msgr2=0x7f19141485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1913e00640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18fc003a80 0x7f19141485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1913e00640 1 -- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1914162fa0 msgr2=0x7f1914163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1913e00640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1914162fa0 0x7f1914163420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1913e00640 1 -- 172.21.15.163:0/4269715487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19140cdb80 con 0x7f19140cd010 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1913e00640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 0x7f19140cd490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1900001df0 tx=0x7f190000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900005270 con 0x7f19140cd010 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1900019030 con 0x7f19140cd010 2024-07-26T06:17:11.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900015bc0 con 0x7f19140cd010 2024-07-26T06:17:11.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19140cbcb0 con 0x7f19140cd010 2024-07-26T06:17:11.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.771+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19140cc270 con 0x7f19140cd010 2024-07-26T06:17:11.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.773+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1914163960 con 0x7f19140cd010 2024-07-26T06:17:11.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.774+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f190000a640 con 0x7f19140cd010 2024-07-26T06:17:11.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.775+0000 7f1910c00640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18f40764b0 0x7f18f4078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:11.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.775+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f190009f770 con 0x7f19140cd010 2024-07-26T06:17:11.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.775+0000 7f1919000640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18f40764b0 0x7f18f4078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:11.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.777+0000 7f1919000640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18f40764b0 0x7f18f4078970 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1908032000 tx=0x7f1908036040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:11.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:11.779+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1900068e40 con 0x7f19140cd010 2024-07-26T06:17:12.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.026+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1914071530 con 0x7f19140cd010 2024-07-26T06:17:12.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.028+0000 7f1910c00640 1 -- 172.21.15.163:0/4269715487 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f1900068740 con 0x7f19140cd010 2024-07-26T06:17:12.031 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:17:12.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18f40764b0 msgr2=0x7f18f4078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:12.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18f40764b0 0x7f18f4078970 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1908032000 tx=0x7f1908036040 comp rx=0 tx=0).stop 2024-07-26T06:17:12.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 msgr2=0x7f19140cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 0x7f19140cd490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1900001df0 tx=0x7f190000cb60 comp rx=0 tx=0).stop 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 shutdown_connections 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f18f40764b0 0x7f18f4078970 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1914162fa0 0x7f1914163420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19140cd010 0x7f19140cd490 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 --2- 172.21.15.163:0/4269715487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18fc003a80 0x7f19141485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 >> 172.21.15.163:0/4269715487 conn(0x7f19140b0e30 msgr2=0x7f1914150ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.030+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 shutdown_connections 2024-07-26T06:17:12.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:12.031+0000 7f191a400640 1 -- 172.21.15.163:0/4269715487 wait complete. 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:17:12.050 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:17:12.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.051 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:17:12.110 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:17:12.110 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:12.111 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42351sec preferred_lft 42351sec 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:12.112 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:17:12.113 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:17:12.113 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.113 INFO:teuthology.orchestra.run.smithi164.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:12.113 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:17:12.113 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::58b1:99ff:fee0:4877/64 scope link 2024-07-26T06:17:12.113 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:12.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.114 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:17:12.194 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:17:12.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.199 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:17:12.252 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:17:12.253 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:17:12.253 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:17:12.253 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:17:12.253 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:17:12.253 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:17:12.253 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:17:12.253 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:17:12.254 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:17:12.254 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:17:12.254 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:17:12.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:17:12.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:12.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.461 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:17:12.505 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:17:12.505 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:17:12.506 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:17:12.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.507 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:17:12.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:12.636 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:12.690 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:17:12.690 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:12.690 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:12.742 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:17:12.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.742 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:12.877 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:12.877 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:17:12.877 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:17:12.898 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:12.898 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:17:12.898 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:17:12.899 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:12.899 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:12.925 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:12.925 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:17:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:17:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:17:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:17:12.926 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:17:12.927 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:17:12.928 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:17:12.928 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:17:12.928 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:17:12.928 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:17:12.931 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:17:12.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:12.931 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:13.003 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:17:13.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:17:13.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:13.121 INFO:teuthology.orchestra.run.smithi164.stdout:client4889 2024-07-26T06:17:13.121 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:13.122 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:13.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51dec00640 1 -- 172.21.15.163:0/2459853155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51c00040b0 msgr2=0x7f51c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51dec00640 1 --2- 172.21.15.163:0/2459853155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51c00040b0 0x7f51c0004800 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f51d806cf00 tx=0x7f51c0005830 comp rx=0 tx=0).stop 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51de200640 1 -- 172.21.15.163:0/2459853155 reap_dead start 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51dec00640 1 -- 172.21.15.163:0/2459853155 shutdown_connections 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51dec00640 1 -- 172.21.15.163:0/2459853155 >> 172.21.15.163:0/2459853155 conn(0x7f51d80b0e30 msgr2=0x7f51d80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51dec00640 1 -- 172.21.15.163:0/2459853155 shutdown_connections 2024-07-26T06:17:13.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.390+0000 7f51dec00640 1 -- 172.21.15.163:0/2459853155 wait complete. 2024-07-26T06:17:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.392+0000 7f51dec00640 1 Processor -- start 2024-07-26T06:17:13.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.392+0000 7f51dec00640 1 -- start start 2024-07-26T06:17:13.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dec00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51c00040b0 0x7f51d80b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:13.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dec00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 0x7f51d81465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:13.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dec00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51d8148fd0 0x7f51d8146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:13.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dec00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51d80ce120 con 0x7f51d8159ed0 2024-07-26T06:17:13.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dec00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f51d80cdfa0 con 0x7f51c00040b0 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dec00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f51d80b3e20 con 0x7f51d8148fd0 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 0x7f51d81465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 0x7f51d81465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50996/0 (socket says 172.21.15.163:50996) 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 -- 172.21.15.163:0/863641006 learned_addr learned my addr 172.21.15.163:0/863641006 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51de200640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51d8148fd0 0x7f51d8146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dd800640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51c00040b0 0x7f51d80b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 -- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51d8148fd0 msgr2=0x7f51d8146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:13.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51d8148fd0 0x7f51d8146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:13.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 -- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51c00040b0 msgr2=0x7f51d80b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:13.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51c00040b0 0x7f51d80b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:13.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dce00640 1 -- 172.21.15.163:0/863641006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d815a3e0 con 0x7f51d8159ed0 2024-07-26T06:17:13.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.393+0000 7f51dd800640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51c00040b0 0x7f51d80b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:13.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.394+0000 7f51dce00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 0x7f51d81465d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f51c400c3a0 tx=0x7f51c4004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.394+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4031040 con 0x7f51d8159ed0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.394+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51c4006420 con 0x7f51d8159ed0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.394+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4006c90 con 0x7f51d8159ed0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.394+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d815a6c0 con 0x7f51d8159ed0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.394+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d815ac80 con 0x7f51d8159ed0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.395+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d80b4380 con 0x7f51d8159ed0 2024-07-26T06:17:13.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.396+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f51c4034070 con 0x7f51d8159ed0 2024-07-26T06:17:13.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.397+0000 7f51d5600640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51b80764b0 0x7f51b8078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:13.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.397+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f51c40b7d80 con 0x7f51d8159ed0 2024-07-26T06:17:13.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.398+0000 7f51dd800640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51b80764b0 0x7f51b8078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:13.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.399+0000 7f51dd800640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51b80764b0 0x7f51b8078970 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0058c0 tx=0x7f51cc005e70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:13.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.402+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f51c40812b0 con 0x7f51d8159ed0 2024-07-26T06:17:13.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:13.648+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f51d80b3aa0 con 0x7f51d8159ed0 2024-07-26T06:17:14.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.373+0000 7f51d5600640 1 -- 172.21.15.163:0/863641006 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v36) ==== 136+0+0 (secure 0 0 0) 0x7f51c4080bb0 con 0x7f51d8159ed0 2024-07-26T06:17:14.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51b80764b0 msgr2=0x7f51b8078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51b80764b0 0x7f51b8078970 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0058c0 tx=0x7f51cc005e70 comp rx=0 tx=0).stop 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 msgr2=0x7f51d81465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 0x7f51d81465d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f51c400c3a0 tx=0x7f51c4004bc0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 shutdown_connections 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f51b80764b0 0x7f51b8078970 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51d8148fd0 0x7f51d8146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51d8159ed0 0x7f51d81465d0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 --2- 172.21.15.163:0/863641006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51c00040b0 0x7f51d80b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.377+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 >> 172.21.15.163:0/863641006 conn(0x7f51d80b0e30 msgr2=0x7f51d814f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:14.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.378+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 shutdown_connections 2024-07-26T06:17:14.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.378+0000 7f51dec00640 1 -- 172.21.15.163:0/863641006 wait complete. 2024-07-26T06:17:14.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-07-26T06:17:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 -- 172.21.15.163:0/781879052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 msgr2=0x7fdc6c0b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 --2- 172.21.15.163:0/781879052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0b4bd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdc54005fd0 tx=0x7fdc5402a4a0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 -- 172.21.15.163:0/781879052 shutdown_connections 2024-07-26T06:17:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 --2- 172.21.15.163:0/781879052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc6c073820 0x7fdc6c073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 --2- 172.21.15.163:0/781879052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0b4bd0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 -- 172.21.15.163:0/781879052 >> 172.21.15.163:0/781879052 conn(0x7fdc6c0b0e30 msgr2=0x7fdc6c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 -- 172.21.15.163:0/781879052 shutdown_connections 2024-07-26T06:17:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.605+0000 7fdc6b200640 1 -- 172.21.15.163:0/781879052 wait complete. 2024-07-26T06:17:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.607+0000 7fdc6b200640 1 Processor -- start 2024-07-26T06:17:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.608+0000 7fdc6b200640 1 -- start start 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc6c073820 0x7fdc6c1535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc6c0b3bd0 0x7fdc6c0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c15a1d0 con 0x7fdc6c0ae870 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c073380 con 0x7fdc6c0b3bd0 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6c15a350 con 0x7fdc6c073820 2024-07-26T06:17:14.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc61400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc61400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:51016/0 (socket says 172.21.15.163:51016) 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc61400640 1 -- 172.21.15.163:0/1583106000 learned_addr learned my addr 172.21.15.163:0/1583106000 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc69e00640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc6c073820 0x7fdc6c1535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.609+0000 7fdc6a800640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc6c0b3bd0 0x7fdc6c0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc61400640 1 -- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc6c073820 msgr2=0x7fdc6c1535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc61400640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc6c073820 0x7fdc6c1535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc61400640 1 -- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc6c0b3bd0 msgr2=0x7fdc6c0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc61400640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc6c0b3bd0 0x7fdc6c0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc61400640 1 -- 172.21.15.163:0/1583106000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c1f7540 con 0x7fdc6c0ae870 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc61400640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0cc430 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdc54000c00 tx=0x7fdc540039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5403f070 con 0x7fdc6c0ae870 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc54004890 con 0x7fdc6c0ae870 2024-07-26T06:17:14.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54035030 con 0x7fdc6c0ae870 2024-07-26T06:17:14.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c1f7820 con 0x7fdc6c0ae870 2024-07-26T06:17:14.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.610+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1f7de0 con 0x7fdc6c0ae870 2024-07-26T06:17:14.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.612+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc6c14d1f0 con 0x7fdc6c0ae870 2024-07-26T06:17:14.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.613+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7fdc54042070 con 0x7fdc6c0ae870 2024-07-26T06:17:14.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.614+0000 7fdc62a00640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc48076480 0x7fdc48078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:14.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.614+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fdc540b8a30 con 0x7fdc6c0ae870 2024-07-26T06:17:14.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.614+0000 7fdc69e00640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc48076480 0x7fdc48078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:14.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.615+0000 7fdc69e00640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc48076480 0x7fdc48078940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c006fd0 tx=0x7fdc5c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:14.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.618+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc6c14d1f0 con 0x7fdc6c0ae870 2024-07-26T06:17:14.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.854+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fdc6c0b26d0 con 0x7fdc48076480 2024-07-26T06:17:14.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.858+0000 7fdc62a00640 1 -- 172.21.15.163:0/1583106000 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fdc6c0b26d0 con 0x7fdc48076480 2024-07-26T06:17:14.859 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:17:14.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:14.859 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephfs" 2024-07-26T06:17:14.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:14.859 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:17:14.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.860+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc48076480 msgr2=0x7fdc48078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.860+0000 7fdc6b200640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc48076480 0x7fdc48078940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c006fd0 tx=0x7fdc5c008040 comp rx=0 tx=0).stop 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 msgr2=0x7fdc6c0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0cc430 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdc54000c00 tx=0x7fdc540039e0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 shutdown_connections 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc48076480 0x7fdc48078940 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc6c0b3bd0 0x7fdc6c0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc6c0ae870 0x7fdc6c0cc430 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 --2- 172.21.15.163:0/1583106000 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc6c073820 0x7fdc6c1535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:14.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 >> 172.21.15.163:0/1583106000 conn(0x7fdc6c0b0e30 msgr2=0x7fdc6c156f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:14.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 shutdown_connections 2024-07-26T06:17:14.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:14.861+0000 7fdc6b200640 1 -- 172.21.15.163:0/1583106000 wait complete. 2024-07-26T06:17:14.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:17:15.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 -- 172.21.15.163:0/3549731644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d540b53b0 msgr2=0x7f6d540b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 --2- 172.21.15.163:0/3549731644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d540b53b0 0x7f6d540b5800 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6d54075ba0 tx=0x7f6d4402aac0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 -- 172.21.15.163:0/3549731644 shutdown_connections 2024-07-26T06:17:15.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 --2- 172.21.15.163:0/3549731644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d540b53b0 0x7f6d540b5800 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 --2- 172.21.15.163:0/3549731644 >> v2:172.21.15.164:3300/0 conn(0x7f6d5415d650 0x7f6d540d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 -- 172.21.15.163:0/3549731644 >> 172.21.15.163:0/3549731644 conn(0x7f6d540d36b0 msgr2=0x7f6d540d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:15.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 -- 172.21.15.163:0/3549731644 shutdown_connections 2024-07-26T06:17:15.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.093+0000 7f6d5c400640 1 -- 172.21.15.163:0/3549731644 wait complete. 2024-07-26T06:17:15.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.096+0000 7f6d5c400640 1 Processor -- start 2024-07-26T06:17:15.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.096+0000 7f6d5c400640 1 -- start start 2024-07-26T06:17:15.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5c400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d5406c270 0x7f6d5415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5c400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d540b53b0 0x7f6d54156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5c400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 0x7f6d54160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5c400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d540d2050 con 0x7f6d5415d650 2024-07-26T06:17:15.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5c400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6d540d1ed0 con 0x7f6d540b53b0 2024-07-26T06:17:15.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5c400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6d540b7730 con 0x7f6d5406c270 2024-07-26T06:17:15.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 0x7f6d54160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 0x7f6d54160500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38756/0 (socket says 172.21.15.163:38756) 2024-07-26T06:17:15.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.097+0000 7f6d5a400640 1 -- 172.21.15.163:0/2178561530 learned_addr learned my addr 172.21.15.163:0/2178561530 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:15.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d59a00640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d5406c270 0x7f6d5415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d59000640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d540b53b0 0x7f6d54156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d5a400640 1 -- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d5406c270 msgr2=0x7f6d5415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d5a400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d5406c270 0x7f6d5415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d5a400640 1 -- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d540b53b0 msgr2=0x7f6d54156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d5a400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d540b53b0 0x7f6d54156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d5a400640 1 -- 172.21.15.163:0/2178561530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d541587a0 con 0x7f6d5415d650 2024-07-26T06:17:15.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d5a400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 0x7f6d54160500 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00c410 tx=0x7f6d4c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:15.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c011790 con 0x7f6d5415d650 2024-07-26T06:17:15.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d4c017070 con 0x7f6d5415d650 2024-07-26T06:17:15.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.098+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c011d00 con 0x7f6d5415d650 2024-07-26T06:17:15.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.099+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d54158a80 con 0x7f6d5415d650 2024-07-26T06:17:15.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.099+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5415dd40 con 0x7f6d5415d650 2024-07-26T06:17:15.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.100+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d54158fa0 con 0x7f6d5415d650 2024-07-26T06:17:15.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.101+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f6d4c01c460 con 0x7f6d5415d650 2024-07-26T06:17:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.102+0000 7f6d51600640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6d38076480 0x7f6d38078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.102+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f6d4c097800 con 0x7f6d5415d650 2024-07-26T06:17:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.103+0000 7f6d59a00640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6d38076480 0x7f6d38078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.104+0000 7f6d59a00640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6d38076480 0x7f6d38078940 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6d48005d70 tx=0x7f6d48005d00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.106+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d54158fa0 con 0x7f6d5415d650 2024-07-26T06:17:15.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.310+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f6d5415df40 con 0x7f6d5415d650 2024-07-26T06:17:15.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.311+0000 7f6d51600640 1 -- 172.21.15.163:0/2178561530 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f6d5415df40 con 0x7f6d5415d650 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.313+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6d38076480 msgr2=0x7f6d38078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6d38076480 0x7f6d38078940 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6d48005d70 tx=0x7f6d48005d00 comp rx=0 tx=0).stop 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 msgr2=0x7f6d54160500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 0x7f6d54160500 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00c410 tx=0x7f6d4c00c8d0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 shutdown_connections 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6d38076480 0x7f6d38078940 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d5415d650 0x7f6d54160500 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d540b53b0 0x7f6d54156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 --2- 172.21.15.163:0/2178561530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d5406c270 0x7f6d5415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 >> 172.21.15.163:0/2178561530 conn(0x7f6d540d36b0 msgr2=0x7f6d5415b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:15.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 shutdown_connections 2024-07-26T06:17:15.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.314+0000 7f6d5c400640 1 -- 172.21.15.163:0/2178561530 wait complete. 2024-07-26T06:17:15.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:15.335 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol84520) 2024-07-26T06:17:15.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:15.426 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-07-26T06:17:15.482 INFO:teuthology.orchestra.run.smithi163.stdout:cephfs_data 2024-07-26T06:17:15.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:15.483 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/_nogroup/subvol84520) 2024-07-26T06:17:15.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol39231 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306d800640 1 -- 172.21.15.163:0/3967186479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f304c004800 msgr2=0x7f304c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306d800640 1 --2- 172.21.15.163:0/3967186479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f304c004800 0x7f304c004c80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3050006fd0 tx=0x7f304c005830 comp rx=0 tx=0).stop 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306ce00640 1 -- 172.21.15.163:0/3967186479 reap_dead start 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306d800640 1 -- 172.21.15.163:0/3967186479 shutdown_connections 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306d800640 1 -- 172.21.15.163:0/3967186479 >> 172.21.15.163:0/3967186479 conn(0x7f30680b0e30 msgr2=0x7f30680b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306d800640 1 -- 172.21.15.163:0/3967186479 shutdown_connections 2024-07-26T06:17:15.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.784+0000 7f306d800640 1 -- 172.21.15.163:0/3967186479 wait complete. 2024-07-26T06:17:15.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.786+0000 7f306d800640 1 Processor -- start 2024-07-26T06:17:15.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.786+0000 7f306d800640 1 -- start start 2024-07-26T06:17:15.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306d800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f304c004800 0x7f30681f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 0x7f3068155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306d800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30681f80f0 0x7f30680ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306d800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3068159430 con 0x7f30681f4d70 2024-07-26T06:17:15.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306d800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f30681592b0 con 0x7f30681f80f0 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306d800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f30681595b0 con 0x7f304c004800 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 0x7f3068155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 0x7f3068155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38766/0 (socket says 172.21.15.163:38766) 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 -- 172.21.15.163:0/1800370295 learned_addr learned my addr 172.21.15.163:0/1800370295 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f306ce00640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30681f80f0 0x7f30680ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f3067e00640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f304c004800 0x7f30681f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 -- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f304c004800 msgr2=0x7f30681f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f304c004800 0x7f30681f16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 -- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30681f80f0 msgr2=0x7f30680ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30681f80f0 0x7f30680ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.787+0000 7f305f400640 1 -- 172.21.15.163:0/1800370295 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30681f8570 con 0x7f30681f4d70 2024-07-26T06:17:15.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.788+0000 7f305f400640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 0x7f3068155560 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f305002b130 tx=0x7f3050004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:15.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.788+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050031040 con 0x7f30681f4d70 2024-07-26T06:17:15.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.788+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3050006440 con 0x7f30681f4d70 2024-07-26T06:17:15.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.788+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050006ca0 con 0x7f30681f4d70 2024-07-26T06:17:15.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.788+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3068155bf0 con 0x7f30681f4d70 2024-07-26T06:17:15.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.788+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30681f4570 con 0x7f30681f4d70 2024-07-26T06:17:15.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.790+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f306815ba90 con 0x7f30681f4d70 2024-07-26T06:17:15.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.791+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 9) ==== 98354+0+0 (secure 0 0 0) 0x7f3050034070 con 0x7f30681f4d70 2024-07-26T06:17:15.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.792+0000 7f3064c00640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30440764b0 0x7f3044078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:15.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.792+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f30500b7b20 con 0x7f30681f4d70 2024-07-26T06:17:15.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.792+0000 7f3067e00640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30440764b0 0x7f3044078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:15.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.793+0000 7f3067e00640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30440764b0 0x7f3044078970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f306814ebf0 tx=0x7f3058006970 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:15.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:15.796+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f306815ba90 con 0x7f30681f4d70 2024-07-26T06:17:16.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.033+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol39231", "target": ["mon-mgr", ""]}) -- 0x7f30681f4080 con 0x7f30440764b0 2024-07-26T06:17:16.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:16.045+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:16.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:16.045+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:16.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:16.045+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:16.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:16.045+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:16.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:16.045+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:16.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.113+0000 7f3064c00640 1 -- 172.21.15.163:0/1800370295 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30681f4080 con 0x7f30440764b0 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30440764b0 msgr2=0x7f3044078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30440764b0 0x7f3044078970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f306814ebf0 tx=0x7f3058006970 comp rx=0 tx=0).stop 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 msgr2=0x7f3068155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 0x7f3068155560 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f305002b130 tx=0x7f3050004df0 comp rx=0 tx=0).stop 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 shutdown_connections 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f30440764b0 0x7f3044078970 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:16.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30681f80f0 0x7f30680ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f30681f4d70 0x7f3068155560 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 --2- 172.21.15.163:0/1800370295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f304c004800 0x7f30681f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.115+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 >> 172.21.15.163:0/1800370295 conn(0x7f30680b0e30 msgr2=0x7f3068147770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.116+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 shutdown_connections 2024-07-26T06:17:16.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:16.116+0000 7f306d800640 1 -- 172.21.15.163:0/1800370295 wait complete. 2024-07-26T06:17:16.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:16.138 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cp ./volumes/_nogroup/subvol39231/.meta ./volumes/_nogroup/subvol84520) 2024-07-26T06:17:18.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol84520 2024-07-26T06:17:18.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 -- 172.21.15.163:0/586799650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f8415d310 msgr2=0x7f1f840b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 --2- 172.21.15.163:0/586799650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f8415d310 0x7f1f840b4d30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1f840d3280 tx=0x7f1f7402a4d0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 -- 172.21.15.163:0/586799650 shutdown_connections 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 --2- 172.21.15.163:0/586799650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 0x7f1f840b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 --2- 172.21.15.163:0/586799650 >> v2:172.21.15.164:3300/0 conn(0x7f1f840ce960 0x7f1f840b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 --2- 172.21.15.163:0/586799650 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f8415d310 0x7f1f840b4d30 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 -- 172.21.15.163:0/586799650 >> 172.21.15.163:0/586799650 conn(0x7f1f840cfd30 msgr2=0x7f1f840d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 -- 172.21.15.163:0/586799650 shutdown_connections 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.553+0000 7f1f89800640 1 -- 172.21.15.163:0/586799650 wait complete. 2024-07-26T06:17:18.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.555+0000 7f1f89800640 1 Processor -- start 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.555+0000 7f1f89800640 1 -- start start 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f89800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 0x7f1f840ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f89800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 0x7f1f840cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f89800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415d310 0x7f1f841f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f89800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f840d1000 con 0x7f1f840ce960 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f89800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1f840d0e80 con 0x7f1f8415d310 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f89800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1f840d1180 con 0x7f1f840c9ec0 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.556+0000 7f1f83400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 0x7f1f840cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 0x7f1f840cc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38780/0 (socket says 172.21.15.163:38780) 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 -- 172.21.15.163:0/3869196095 learned_addr learned my addr 172.21.15.163:0/3869196095 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83e00640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 0x7f1f840ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:18.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 -- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 msgr2=0x7f1f840ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 0x7f1f840ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 -- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415d310 msgr2=0x7f1f841f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f88e00640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415d310 0x7f1f841f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415d310 0x7f1f841f3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 -- 172.21.15.163:0/3869196095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f840a9d80 con 0x7f1f840ce960 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83e00640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 0x7f1f840ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.557+0000 7f1f83400640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 0x7f1f840cc240 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00b4d0 tx=0x7f1f6c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.558+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c014070 con 0x7f1f840ce960 2024-07-26T06:17:18.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.558+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f6c010960 con 0x7f1f840ce960 2024-07-26T06:17:18.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.558+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c010db0 con 0x7f1f840ce960 2024-07-26T06:17:18.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.558+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f840aa060 con 0x7f1f840ce960 2024-07-26T06:17:18.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.558+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f840aa620 con 0x7f1f840ce960 2024-07-26T06:17:18.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.559+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f840685b0 con 0x7f1f840ce960 2024-07-26T06:17:18.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.560+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) ==== 98405+0+0 (secure 0 0 0) 0x7f1f6c017070 con 0x7f1f840ce960 2024-07-26T06:17:18.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.561+0000 7f1f7be00640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1f60076510 0x7f1f600789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:18.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.561+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f1f6c097a10 con 0x7f1f840ce960 2024-07-26T06:17:18.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.562+0000 7f1f83e00640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1f60076510 0x7f1f600789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:18.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.563+0000 7f1f83e00640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1f60076510 0x7f1f600789d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f74033000 tx=0x7f1f74004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:18.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.566+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f840685b0 con 0x7f1f840ce960 2024-07-26T06:17:18.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.804+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol84520", "target": ["mon-mgr", ""]}) -- 0x7f1f840b2110 con 0x7f1f60076510 2024-07-26T06:17:18.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.889+0000 7f1f7be00640 1 -- 172.21.15.163:0/3869196095 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f1f840b2110 con 0x7f1f60076510 2024-07-26T06:17:18.890 INFO:teuthology.orchestra.run.smithi163.stdout:/volumes/_nogroup/subvol84520 2024-07-26T06:17:18.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1f60076510 msgr2=0x7f1f600789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:18.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1f60076510 0x7f1f600789d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f74033000 tx=0x7f1f74004730 comp rx=0 tx=0).stop 2024-07-26T06:17:18.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 msgr2=0x7f1f840cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:18.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 0x7f1f840cc240 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00b4d0 tx=0x7f1f6c00b990 comp rx=0 tx=0).stop 2024-07-26T06:17:18.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 shutdown_connections 2024-07-26T06:17:18.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1f60076510 0x7f1f600789d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415d310 0x7f1f841f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f840ce960 0x7f1f840cc240 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 --2- 172.21.15.163:0/3869196095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840c9ec0 0x7f1f840ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:18.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 >> 172.21.15.163:0/3869196095 conn(0x7f1f840cfd30 msgr2=0x7f1f840b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:18.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 shutdown_connections 2024-07-26T06:17:18.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:18.893+0000 7f1f89800640 1 -- 172.21.15.163:0/3869196095 wait complete. 2024-07-26T06:17:18.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol84520 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.129+0000 7f4015000640 1 -- 172.21.15.163:0/2069709858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100ce960 msgr2=0x7f40100b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.129+0000 7f4015000640 1 --2- 172.21.15.163:0/2069709858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100ce960 0x7f40100b5270 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009830 tx=0x7f3ff802a480 comp rx=0 tx=0).stop 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.129+0000 7f4015000640 1 -- 172.21.15.163:0/2069709858 shutdown_connections 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.129+0000 7f4015000640 1 --2- 172.21.15.163:0/2069709858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100ce960 0x7f40100b5270 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.129+0000 7f4015000640 1 --2- 172.21.15.163:0/2069709858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f401015d310 0x7f40100b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.129+0000 7f4015000640 1 -- 172.21.15.163:0/2069709858 >> 172.21.15.163:0/2069709858 conn(0x7f40100cfd30 msgr2=0x7f40100d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.130+0000 7f4015000640 1 -- 172.21.15.163:0/2069709858 shutdown_connections 2024-07-26T06:17:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.130+0000 7f4015000640 1 -- 172.21.15.163:0/2069709858 wait complete. 2024-07-26T06:17:19.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.132+0000 7f4015000640 1 Processor -- start 2024-07-26T06:17:19.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.132+0000 7f4015000640 1 -- start start 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.133+0000 7f4015000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 0x7f401014ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.133+0000 7f4015000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40100ce960 0x7f40101f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.133+0000 7f4015000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f401015d310 0x7f40100ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.133+0000 7f4015000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40100b5930 con 0x7f40100c9ec0 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.133+0000 7f4015000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40100b57b0 con 0x7f40100ce960 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.133+0000 7f4015000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f40100b5ab0 con 0x7f401015d310 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 0x7f401014ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 0x7f401014ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38800/0 (socket says 172.21.15.163:38800) 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 -- 172.21.15.163:0/1129258482 learned_addr learned my addr 172.21.15.163:0/1129258482 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:19.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400fe00640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f401015d310 0x7f40100ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f4006a00640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40100ce960 0x7f40101f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 -- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f401015d310 msgr2=0x7f40100ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f401015d310 0x7f40100ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 -- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40100ce960 msgr2=0x7f40101f7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40100ce960 0x7f40101f7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.134+0000 7f400f400640 1 -- 172.21.15.163:0/1129258482 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40101f7f50 con 0x7f40100c9ec0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.135+0000 7f400f400640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 0x7f401014ca00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f400000e790 tx=0x7f400000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.135+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400000eea0 con 0x7f40100c9ec0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.135+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f400000cdb0 con 0x7f40100c9ec0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.135+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401014f730 con 0x7f40100c9ec0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.135+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000010690 con 0x7f40100c9ec0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.135+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401014fcf0 con 0x7f40100c9ec0 2024-07-26T06:17:19.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.137+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40100d26c0 con 0x7f40100c9ec0 2024-07-26T06:17:19.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.138+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) ==== 98405+0+0 (secure 0 0 0) 0x7f4000010830 con 0x7f40100c9ec0 2024-07-26T06:17:19.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.139+0000 7f4007e00640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3fec076510 0x7f3fec0789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.139+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f40000985b0 con 0x7f40100c9ec0 2024-07-26T06:17:19.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.139+0000 7f4006a00640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3fec076510 0x7f3fec0789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.140+0000 7f4006a00640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3fec076510 0x7f3fec0789d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8034000 tx=0x7f3ff8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:19.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.143+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f40100d26c0 con 0x7f40100c9ec0 2024-07-26T06:17:19.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.385+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol84520", "target": ["mon-mgr", ""]}) -- 0x7f40100d0e80 con 0x7f3fec076510 2024-07-26T06:17:19.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.397+0000 7f4007e00640 1 -- 172.21.15.163:0/1129258482 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f40100d0e80 con 0x7f3fec076510 2024-07-26T06:17:19.398 INFO:teuthology.orchestra.run.smithi163.stdout:/volumes/_nogroup/subvol84520 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3fec076510 msgr2=0x7f3fec0789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3fec076510 0x7f3fec0789d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8034000 tx=0x7f3ff8004730 comp rx=0 tx=0).stop 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 msgr2=0x7f401014ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 0x7f401014ca00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f400000e790 tx=0x7f400000ec50 comp rx=0 tx=0).stop 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 shutdown_connections 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3fec076510 0x7f3fec0789d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f401015d310 0x7f40100ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40100ce960 0x7f40101f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 --2- 172.21.15.163:0/1129258482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40100c9ec0 0x7f401014ca00 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 >> 172.21.15.163:0/1129258482 conn(0x7f40100cfd30 msgr2=0x7f401015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:19.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 shutdown_connections 2024-07-26T06:17:19.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.400+0000 7f4015000640 1 -- 172.21.15.163:0/1129258482 wait complete. 2024-07-26T06:17:19.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:19.421 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/2737eb6e9dcb5eb2ed14fa9075f5617d.meta) 2024-07-26T06:17:19.509 INFO:teuthology.orchestra.run.smithi163.stdout:[GLOBAL] 2024-07-26T06:17:19.509 INFO:teuthology.orchestra.run.smithi163.stdout:version = 1 2024-07-26T06:17:19.509 INFO:teuthology.orchestra.run.smithi163.stdout:type = subvolume 2024-07-26T06:17:19.510 INFO:teuthology.orchestra.run.smithi163.stdout:path = /volumes/_nogroup/subvol84520 2024-07-26T06:17:19.510 INFO:teuthology.orchestra.run.smithi163.stdout:state = complete 2024-07-26T06:17:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:19.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume authorize cephfs subvol84520 alice 2024-07-26T06:17:19.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.730+0000 7f92bbe00640 1 -- 172.21.15.163:0/3375841373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc06f2f0 msgr2=0x7f92bc1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.730+0000 7f92bbe00640 1 --2- 172.21.15.163:0/3375841373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc06f2f0 0x7f92bc1590a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f92bc071190 tx=0x7f92a402a470 comp rx=0 tx=0).stop 2024-07-26T06:17:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.731+0000 7f92bbe00640 1 -- 172.21.15.163:0/3375841373 shutdown_connections 2024-07-26T06:17:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.731+0000 7f92bbe00640 1 --2- 172.21.15.163:0/3375841373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc0cd3a0 0x7f92bc0cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.731+0000 7f92bbe00640 1 --2- 172.21.15.163:0/3375841373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc06f2f0 0x7f92bc1590a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.731+0000 7f92bbe00640 1 -- 172.21.15.163:0/3375841373 >> 172.21.15.163:0/3375841373 conn(0x7f92bc0b0e30 msgr2=0x7f92bc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.731+0000 7f92bbe00640 1 -- 172.21.15.163:0/3375841373 shutdown_connections 2024-07-26T06:17:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.731+0000 7f92bbe00640 1 -- 172.21.15.163:0/3375841373 wait complete. 2024-07-26T06:17:19.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.733+0000 7f92bbe00640 1 Processor -- start 2024-07-26T06:17:19.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.733+0000 7f92bbe00640 1 -- start start 2024-07-26T06:17:19.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bbe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc06f2f0 0x7f92bc142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bbe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc0ccdc0 0x7f92bc144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 0x7f92bc1450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bbe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc0b22b0 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bbe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc0b2130 con 0x7f92bc06f2f0 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bbe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f92bc0b2430 con 0x7f92bc0ccdc0 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 0x7f92bc1450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 0x7f92bc1450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38820/0 (socket says 172.21.15.163:38820) 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.734+0000 7f92bb400640 1 -- 172.21.15.163:0/522709864 learned_addr learned my addr 172.21.15.163:0/522709864 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:19.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92baa00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc06f2f0 0x7f92bc142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92b2000640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc0ccdc0 0x7f92bc144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92bb400640 1 -- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc0ccdc0 msgr2=0x7f92bc144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92bb400640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc0ccdc0 0x7f92bc144730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92bb400640 1 -- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc06f2f0 msgr2=0x7f92bc142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:19.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92bb400640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc06f2f0 0x7f92bc142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:19.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92bb400640 1 -- 172.21.15.163:0/522709864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc1439b0 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92bb400640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 0x7f92bc1450c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f92b400e8b0 tx=0x7f92b400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.735+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4004d20 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.736+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92b400ca60 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.736+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4005020 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.736+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92bc145c10 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.736+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92bc143f30 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.737+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92bc1590a0 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.738+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) ==== 98405+0+0 (secure 0 0 0) 0x7f92b40051c0 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.739+0000 7f92b3400640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9298076510 0x7f92980789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:19.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.739+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f92b4098430 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.740+0000 7f92baa00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9298076510 0x7f92980789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:19.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.741+0000 7f92baa00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9298076510 0x7f92980789d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f92ac004ea0 tx=0x7f92ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:19.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.743+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f92b4016030 con 0x7f92bc0cd3a0 2024-07-26T06:17:19.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:19.978+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvol84520", "auth_id": "alice", "target": ["mon-mgr", ""]}) -- 0x7f92bc14a900 con 0x7f9298076510 2024-07-26T06:17:20.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:Creating meta for ID alice with tenant None 2024-07-26T06:17:20.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.087+0000 7f92b3400640 1 -- 172.21.15.163:0/522709864 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f92bc14a900 con 0x7f9298076510 2024-07-26T06:17:20.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9298076510 msgr2=0x7f92980789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9298076510 0x7f92980789d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f92ac004ea0 tx=0x7f92ac009040 comp rx=0 tx=0).stop 2024-07-26T06:17:20.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 msgr2=0x7f92bc1450c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 0x7f92bc1450c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f92b400e8b0 tx=0x7f92b400ed70 comp rx=0 tx=0).stop 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 shutdown_connections 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9298076510 0x7f92980789d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92bc0cd3a0 0x7f92bc1450c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc0ccdc0 0x7f92bc144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 --2- 172.21.15.163:0/522709864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc06f2f0 0x7f92bc142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 >> 172.21.15.163:0/522709864 conn(0x7f92bc0b0e30 msgr2=0x7f92bc067dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 shutdown_connections 2024-07-26T06:17:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.091+0000 7f92bbe00640 1 -- 172.21.15.163:0/522709864 wait complete. 2024-07-26T06:17:20.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get client.alice --format=json-pretty 2024-07-26T06:17:20.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 -- 172.21.15.163:0/4261107130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27840b1e80 msgr2=0x7f27840b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 --2- 172.21.15.163:0/4261107130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27840b1e80 0x7f27840b6a00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f27840690b0 tx=0x7f277c02a8f0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 -- 172.21.15.163:0/4261107130 shutdown_connections 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 --2- 172.21.15.163:0/4261107130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27840b1e80 0x7f27840b6a00 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 --2- 172.21.15.163:0/4261107130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27840ce830 0x7f27840b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 --2- 172.21.15.163:0/4261107130 >> v2:172.21.15.164:3300/0 conn(0x7f278415d5a0 0x7f27840ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 -- 172.21.15.163:0/4261107130 >> 172.21.15.163:0/4261107130 conn(0x7f27840cfda0 msgr2=0x7f27840d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 -- 172.21.15.163:0/4261107130 shutdown_connections 2024-07-26T06:17:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.319+0000 7f2789600640 1 -- 172.21.15.163:0/4261107130 wait complete. 2024-07-26T06:17:20.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.321+0000 7f2789600640 1 Processor -- start 2024-07-26T06:17:20.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.321+0000 7f2789600640 1 -- start start 2024-07-26T06:17:20.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2789600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840b1e80 0x7f27840c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2789600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27840ce830 0x7f278414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2789600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 0x7f278415ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2789600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f27840b4da0 con 0x7f278415d5a0 2024-07-26T06:17:20.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2789600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f27840b2630 con 0x7f27840b1e80 2024-07-26T06:17:20.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2789600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f27840b4f20 con 0x7f27840ce830 2024-07-26T06:17:20.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2788c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 0x7f278415ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:20.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.322+0000 7f2788c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 0x7f278415ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38852/0 (socket says 172.21.15.163:38852) 2024-07-26T06:17:20.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 -- 172.21.15.163:0/3943484506 learned_addr learned my addr 172.21.15.163:0/3943484506 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:20.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f277b400640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27840ce830 0x7f278414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2783e00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840b1e80 0x7f27840c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 -- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27840ce830 msgr2=0x7f278414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27840ce830 0x7f278414e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 -- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840b1e80 msgr2=0x7f27840c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840b1e80 0x7f27840c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 -- 172.21.15.163:0/3943484506 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278415f7f0 con 0x7f278415d5a0 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2783e00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840b1e80 0x7f27840c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.323+0000 7f2788c00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 0x7f278415ef10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f277c033040 tx=0x7f277c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.324+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c004040 con 0x7f278415d5a0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.324+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f277c02d070 con 0x7f278415d5a0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.324+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278415fad0 con 0x7f278415d5a0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.324+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c004440 con 0x7f278415d5a0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.324+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27841526d0 con 0x7f278415d5a0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.325+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278415fff0 con 0x7f278415d5a0 2024-07-26T06:17:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.326+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) ==== 98405+0+0 (secure 0 0 0) 0x7f277c0045e0 con 0x7f278415d5a0 2024-07-26T06:17:20.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.327+0000 7f2780c00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2760076510 0x7f27600789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.328+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f277c0b9660 con 0x7f278415d5a0 2024-07-26T06:17:20.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.328+0000 7f2783e00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2760076510 0x7f27600789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.329+0000 7f2783e00640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2760076510 0x7f27600789d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2774009aa0 tx=0x7f2774005db0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:20.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.332+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f278415fff0 con 0x7f278415d5a0 2024-07-26T06:17:20.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.615+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"} v 0) -- 0x7f27840ce2f0 con 0x7f278415d5a0 2024-07-26T06:17:20.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.616+0000 7f2780c00640 1 -- 172.21.15.163:0/3943484506 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"}]=0 v13) ==== 107+0+280 (secure 0 0 0) 0x7f277c082bc0 con 0x7f278415d5a0 2024-07-26T06:17:20.616 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.alice", 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBwP6NmeYMJAhAA1U3hd/zPtWuZfG50TBHX5w==", 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol84520", 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw pool=cephfs_data" 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:20.617 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:17:20.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2760076510 msgr2=0x7f27600789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2760076510 0x7f27600789d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2774009aa0 tx=0x7f2774005db0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 msgr2=0x7f278415ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 0x7f278415ef10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f277c033040 tx=0x7f277c003ba0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 shutdown_connections 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2760076510 0x7f27600789d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f278415d5a0 0x7f278415ef10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27840ce830 0x7f278414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 --2- 172.21.15.163:0/3943484506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840b1e80 0x7f27840c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 >> 172.21.15.163:0/3943484506 conn(0x7f27840cfda0 msgr2=0x7f27840b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.618+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 shutdown_connections 2024-07-26T06:17:20.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.619+0000 7f2789600640 1 -- 172.21.15.163:0/3943484506 wait complete. 2024-07-26T06:17:20.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol84520 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2927738484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 msgr2=0x7fc594003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2927738484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc594003e70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000baf0 tx=0x7fc594005830 comp rx=0 tx=0).stop 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2927738484 shutdown_connections 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2927738484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc594003e70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2927738484 >> 172.21.15.163:0/2927738484 conn(0x7fc5ac0b0e30 msgr2=0x7fc5ac0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2927738484 shutdown_connections 2024-07-26T06:17:20.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.846+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2927738484 wait complete. 2024-07-26T06:17:20.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.847+0000 7fc5b2e00640 1 Processor -- start 2024-07-26T06:17:20.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 -- start start 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc5ac14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5ac144790 0x7fc5ac145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5ac146510 0x7fc5ac148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0ce120 con 0x7fc594003a50 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0cdfa0 con 0x7fc5ac144790 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.848+0000 7fc5b2e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac0736c0 con 0x7fc5ac146510 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc5ac14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc5ac14b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38870/0 (socket says 172.21.15.163:38870) 2024-07-26T06:17:20.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 -- 172.21.15.163:0/2660307401 learned_addr learned my addr 172.21.15.163:0/2660307401 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:20.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 -- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5ac146510 msgr2=0x7fc5ac148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5ac146510 0x7fc5ac148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 -- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5ac144790 msgr2=0x7fc5ac145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:20.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5ac144790 0x7fc5ac145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:20.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 -- 172.21.15.163:0/2660307401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac147a30 con 0x7fc594003a50 2024-07-26T06:17:20.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.849+0000 7fc5b1a00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc5ac14b280 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000bc10 tx=0x7fc5a0032720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:20.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.850+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0032d20 con 0x7fc594003a50 2024-07-26T06:17:20.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.850+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5a00453f0 con 0x7fc594003a50 2024-07-26T06:17:20.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.850+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac146990 con 0x7fc594003a50 2024-07-26T06:17:20.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.850+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a00457e0 con 0x7fc594003a50 2024-07-26T06:17:20.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.850+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac146f50 con 0x7fc594003a50 2024-07-26T06:17:20.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.851+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac152370 con 0x7fc594003a50 2024-07-26T06:17:20.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.852+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) ==== 98405+0+0 (secure 0 0 0) 0x7fc5a0045980 con 0x7fc594003a50 2024-07-26T06:17:20.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.853+0000 7fc5a9600640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc58c076510 0x7fc58c0789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:20.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.853+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fc5a00b8ed0 con 0x7fc594003a50 2024-07-26T06:17:20.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.854+0000 7fc5b1000640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc58c076510 0x7fc58c0789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:20.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.855+0000 7fc5b1000640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc58c076510 0x7fc58c0789d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac15d1c0 tx=0x7fc598006d90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:20.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:20.858+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc5ac152370 con 0x7fc594003a50 2024-07-26T06:17:21.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.091+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol84520", "target": ["mon-mgr", ""]}) -- 0x7fc5ac0b34f0 con 0x7fc58c076510 2024-07-26T06:17:21.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.143+0000 7fc5a9600640 1 -- 172.21.15.163:0/2660307401 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc5ac0b34f0 con 0x7fc58c076510 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.145+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc58c076510 msgr2=0x7fc58c0789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.145+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc58c076510 0x7fc58c0789d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac15d1c0 tx=0x7fc598006d90 comp rx=0 tx=0).stop 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.145+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 msgr2=0x7fc5ac14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.145+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc5ac14b280 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000bc10 tx=0x7fc5a0032720 comp rx=0 tx=0).stop 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 shutdown_connections 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc58c076510 0x7fc58c0789d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5ac146510 0x7fc5ac148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5ac144790 0x7fc5ac145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 --2- 172.21.15.163:0/2660307401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc594003a50 0x7fc5ac14b280 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 >> 172.21.15.163:0/2660307401 conn(0x7fc5ac0b0e30 msgr2=0x7fc5ac152c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 shutdown_connections 2024-07-26T06:17:21.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.146+0000 7fc5b2e00640 1 -- 172.21.15.163:0/2660307401 wait complete. 2024-07-26T06:17:21.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.153+0000 7f6cf6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.153+0000 7f6cf6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.153+0000 7f6cf6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.153+0000 7f6cf6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.153+0000 7f6cf6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol39231 2024-07-26T06:17:21.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.177+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.177+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.177+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.178+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:17:21.178+0000 7f6cf5400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:17:21.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.379+0000 7fdcb9200640 1 -- 172.21.15.163:0/3738527306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 msgr2=0x7fdc9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:21.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.379+0000 7fdcb9200640 1 --2- 172.21.15.163:0/3738527306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdc9800b9c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40b1ae0 tx=0x7fdc98005830 comp rx=0 tx=0).stop 2024-07-26T06:17:21.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.379+0000 7fdcb3e00640 1 -- 172.21.15.163:0/3738527306 reap_dead start 2024-07-26T06:17:21.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.379+0000 7fdcb9200640 1 -- 172.21.15.163:0/3738527306 shutdown_connections 2024-07-26T06:17:21.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.379+0000 7fdcb9200640 1 -- 172.21.15.163:0/3738527306 >> 172.21.15.163:0/3738527306 conn(0x7fdcb40b0e30 msgr2=0x7fdcb40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:21.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.380+0000 7fdcb9200640 1 -- 172.21.15.163:0/3738527306 shutdown_connections 2024-07-26T06:17:21.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.380+0000 7fdcb9200640 1 -- 172.21.15.163:0/3738527306 wait complete. 2024-07-26T06:17:21.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.381+0000 7fdcb9200640 1 Processor -- start 2024-07-26T06:17:21.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.381+0000 7fdcb9200640 1 -- start start 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb9200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdcb41527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb9200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcb40cae20 0x7fdcb4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb9200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcb4157550 0x7fdcb41e1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb9200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb40cd6b0 con 0x7fdc98005180 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb9200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb40cd530 con 0x7fdcb40cae20 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb9200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb40cd830 con 0x7fdcb4157550 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdcb41527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdcb41527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38890/0 (socket says 172.21.15.163:38890) 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.382+0000 7fdcb3400640 1 -- 172.21.15.163:0/158589378 learned_addr learned my addr 172.21.15.163:0/158589378 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb3400640 1 -- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcb4157550 msgr2=0x7fdcb41e1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb3400640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcb4157550 0x7fdcb41e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb3400640 1 -- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcb40cae20 msgr2=0x7fdcb4156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:21.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb3400640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcb40cae20 0x7fdcb4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:21.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb3400640 1 -- 172.21.15.163:0/158589378 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb41558d0 con 0x7fdc98005180 2024-07-26T06:17:21.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb3400640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdcb41527e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdca400eb90 tx=0x7fdca400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:21.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4005220 con 0x7fdc98005180 2024-07-26T06:17:21.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdca400ce90 con 0x7fdc98005180 2024-07-26T06:17:21.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb41e2520 con 0x7fdc98005180 2024-07-26T06:17:21.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.383+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca40047a0 con 0x7fdc98005180 2024-07-26T06:17:21.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.384+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb41e2ac0 con 0x7fdc98005180 2024-07-26T06:17:21.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.385+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb40cbad0 con 0x7fdc98005180 2024-07-26T06:17:21.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.386+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 10) ==== 98405+0+0 (secure 0 0 0) 0x7fdca4004940 con 0x7fdc98005180 2024-07-26T06:17:21.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.387+0000 7fdcab400640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc94076540 0x7fdc94078a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:21.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.387+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fdca4098430 con 0x7fdc98005180 2024-07-26T06:17:21.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.387+0000 7fdcb2a00640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc94076540 0x7fdc94078a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:21.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.388+0000 7fdcb2a00640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc94076540 0x7fdc94078a00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdca0000fb0 tx=0x7fdca0001c50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:21.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.391+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdca4013070 con 0x7fdc98005180 2024-07-26T06:17:21.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:21.630+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol39231", "target": ["mon-mgr", ""]}) -- 0x7fdcb406d970 con 0x7fdc94076540 2024-07-26T06:17:22.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:22.441+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fdca405d4a0 con 0x7fdc98005180 2024-07-26T06:17:26.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:17:26.777 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:17:28.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.346+0000 7fdcab400640 1 -- 172.21.15.163:0/158589378 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdcb406d970 con 0x7fdc94076540 2024-07-26T06:17:28.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc94076540 msgr2=0x7fdc94078a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc94076540 0x7fdc94078a00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdca0000fb0 tx=0x7fdca0001c50 comp rx=0 tx=0).stop 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 msgr2=0x7fdcb41527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdcb41527e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdca400eb90 tx=0x7fdca400c740 comp rx=0 tx=0).stop 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 shutdown_connections 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fdc94076540 0x7fdc94078a00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcb4157550 0x7fdcb41e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcb40cae20 0x7fdcb4156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 --2- 172.21.15.163:0/158589378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98005180 0x7fdcb41527e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 >> 172.21.15.163:0/158589378 conn(0x7fdcb40b0e30 msgr2=0x7fdcb4159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 shutdown_connections 2024-07-26T06:17:28.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28.349+0000 7fdcb9200640 1 -- 172.21.15.163:0/158589378 wait complete. 2024-07-26T06:17:28.373 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-07-26T06:17:28.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:17:28.486 INFO:teuthology.orchestra.run.smithi163.stdout:2 2024-07-26T06:17:28.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:28.496 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:17:28.497 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:17:28.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:17:28.740 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:17:28.740 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:17:28.740 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:28.740 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:28.799 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:28.799 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:28.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:17:29.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.058+0000 7f6303200640 1 -- 172.21.15.163:0/1088750632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 msgr2=0x7f63040cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.058+0000 7f6303200640 1 --2- 172.21.15.163:0/1088750632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f63040cdd90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f63040ced10 tx=0x7f62f402a530 comp rx=0 tx=0).stop 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.059+0000 7f6303200640 1 -- 172.21.15.163:0/1088750632 shutdown_connections 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.059+0000 7f6303200640 1 --2- 172.21.15.163:0/1088750632 >> v2:172.21.15.164:3300/0 conn(0x7f6304071c40 0x7f63040b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.059+0000 7f6303200640 1 --2- 172.21.15.163:0/1088750632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f63040cdd90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.059+0000 7f6303200640 1 -- 172.21.15.163:0/1088750632 >> 172.21.15.163:0/1088750632 conn(0x7f63040b0e30 msgr2=0x7f63040b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.059+0000 7f6303200640 1 -- 172.21.15.163:0/1088750632 shutdown_connections 2024-07-26T06:17:29.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.059+0000 7f6303200640 1 -- 172.21.15.163:0/1088750632 wait complete. 2024-07-26T06:17:29.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.060+0000 7f6303200640 1 Processor -- start 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 -- start start 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6304071c40 0x7f6304148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63040ce2d0 0x7f630414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f630414d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f63040b4020 con 0x7f63041556f0 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f63040b3ea0 con 0x7f63040ce2d0 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.061+0000 7f6303200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63040b41a0 con 0x7f6304071c40 2024-07-26T06:17:29.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f630414d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f630414d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50434/0 (socket says 172.21.15.163:50434) 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 -- 172.21.15.163:0/3754501975 learned_addr learned my addr 172.21.15.163:0/3754501975 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6301400640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63040ce2d0 0x7f630414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 -- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6304071c40 msgr2=0x7f6304148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6304071c40 0x7f6304148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 -- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63040ce2d0 msgr2=0x7f630414caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63040ce2d0 0x7f630414caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.062+0000 7f6302800640 1 -- 172.21.15.163:0/3754501975 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630414f2d0 con 0x7f63041556f0 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.063+0000 7f6302800640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f630414d030 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00ead0 tx=0x7f62fc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.063+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00cde0 con 0x7f63041556f0 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.063+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62fc004c10 con 0x7f63041556f0 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.063+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc005020 con 0x7f63041556f0 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.063+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630414d6d0 con 0x7f63041556f0 2024-07-26T06:17:29.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.063+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63040cbf70 con 0x7f63041556f0 2024-07-26T06:17:29.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.064+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63040b4730 con 0x7f63041556f0 2024-07-26T06:17:29.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.065+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f62fc0051c0 con 0x7f63041556f0 2024-07-26T06:17:29.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.066+0000 7f62fa000640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62e4076630 0x7f62e4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.066+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f62fc098810 con 0x7f63041556f0 2024-07-26T06:17:29.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.067+0000 7f6301e00640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62e4076630 0x7f62e4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.068+0000 7f6301e00640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62e4076630 0x7f62e4078af0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f62f4034000 tx=0x7f62f40046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:29.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.071+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f630414f2d0 con 0x7f63041556f0 2024-07-26T06:17:29.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.292+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f63040b34f0 con 0x7f63041556f0 2024-07-26T06:17:29.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.293+0000 7f62fa000640 1 -- 172.21.15.163:0/3754501975 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f62fc016030 con 0x7f63041556f0 2024-07-26T06:17:29.294 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:17:29.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62e4076630 msgr2=0x7f62e4078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62e4076630 0x7f62e4078af0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f62f4034000 tx=0x7f62f40046c0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 msgr2=0x7f630414d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f630414d030 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00ead0 tx=0x7f62fc00ef90 comp rx=0 tx=0).stop 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 shutdown_connections 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f62e4076630 0x7f62e4078af0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63041556f0 0x7f630414d030 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63040ce2d0 0x7f630414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 --2- 172.21.15.163:0/3754501975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6304071c40 0x7f6304148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.296+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 >> 172.21.15.163:0/3754501975 conn(0x7f63040b0e30 msgr2=0x7f630415b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.297+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 shutdown_connections 2024-07-26T06:17:29.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.297+0000 7f6303200640 1 -- 172.21.15.163:0/3754501975 wait complete. 2024-07-26T06:17:29.318 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:17:29.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:29.319 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:29.444 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:17:29.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:29.445 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:17:29.496 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:17:29.497 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:29.497 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:29.536 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:29.537 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:29.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 -- 172.21.15.163:0/1398916000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c0cc8e0 msgr2=0x7f383c0ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 --2- 172.21.15.163:0/1398916000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c0cc8e0 0x7f383c0ccd50 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f383c069760 tx=0x7f383402a820 comp rx=0 tx=0).stop 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 -- 172.21.15.163:0/1398916000 shutdown_connections 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 --2- 172.21.15.163:0/1398916000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c0cc8e0 0x7f383c0ccd50 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 --2- 172.21.15.163:0/1398916000 >> v2:172.21.15.164:3300/0 conn(0x7f383c06f350 0x7f383c0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 -- 172.21.15.163:0/1398916000 >> 172.21.15.163:0/1398916000 conn(0x7f383c06d420 msgr2=0x7f383c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:29.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 -- 172.21.15.163:0/1398916000 shutdown_connections 2024-07-26T06:17:29.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.829+0000 7f3841600640 1 -- 172.21.15.163:0/1398916000 wait complete. 2024-07-26T06:17:29.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.831+0000 7f3841600640 1 Processor -- start 2024-07-26T06:17:29.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.831+0000 7f3841600640 1 -- start start 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f3841600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f383c06f350 0x7f383c151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f3841600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f383c0cc8e0 0x7f383c1f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f3841600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 0x7f383c072420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f3841600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f383c0cda00 con 0x7f383c159380 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f3841600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f383c0cd880 con 0x7f383c0cc8e0 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f3841600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f383c0cdb80 con 0x7f383c06f350 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f383b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 0x7f383c072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f383b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 0x7f383c072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50444/0 (socket says 172.21.15.163:50444) 2024-07-26T06:17:29.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.832+0000 7f383b400640 1 -- 172.21.15.163:0/2990535185 learned_addr learned my addr 172.21.15.163:0/2990535185 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383a000640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f383c0cc8e0 0x7f383c1f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383aa00640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f383c06f350 0x7f383c151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383b400640 1 -- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f383c06f350 msgr2=0x7f383c151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383b400640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f383c06f350 0x7f383c151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383b400640 1 -- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f383c0cc8e0 msgr2=0x7f383c1f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383b400640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f383c0cc8e0 0x7f383c1f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383b400640 1 -- 172.21.15.163:0/2990535185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c159bc0 con 0x7f383c159380 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383aa00640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f383c06f350 0x7f383c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f383b400640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 0x7f383c072420 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3834009ca0 tx=0x7f3834032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383403f020 con 0x7f383c159380 2024-07-26T06:17:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3834004040 con 0x7f383c159380 2024-07-26T06:17:29.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.833+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834037040 con 0x7f383c159380 2024-07-26T06:17:29.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.834+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c159ea0 con 0x7f383c159380 2024-07-26T06:17:29.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.834+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c15a460 con 0x7f383c159380 2024-07-26T06:17:29.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.835+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c0b2f90 con 0x7f383c159380 2024-07-26T06:17:29.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.836+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f3834042020 con 0x7f383c159380 2024-07-26T06:17:29.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.837+0000 7f3832a00640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381c076660 0x7f381c078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:29.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.837+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f38340b8130 con 0x7f383c159380 2024-07-26T06:17:29.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.837+0000 7f383aa00640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381c076660 0x7f381c078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:29.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.838+0000 7f383aa00640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381c076660 0x7f381c078b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f382c00cb50 tx=0x7f382c005e50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:29.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:29.841+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3834049070 con 0x7f383c159380 2024-07-26T06:17:30.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.068+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f383c0b4420 con 0x7f383c159380 2024-07-26T06:17:30.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.069+0000 7f3832a00640 1 -- 172.21.15.163:0/2990535185 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f3834081530 con 0x7f383c159380 2024-07-26T06:17:30.070 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:17:30.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381c076660 msgr2=0x7f381c078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381c076660 0x7f381c078b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f382c00cb50 tx=0x7f382c005e50 comp rx=0 tx=0).stop 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 msgr2=0x7f383c072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 0x7f383c072420 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3834009ca0 tx=0x7f3834032c90 comp rx=0 tx=0).stop 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 shutdown_connections 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f381c076660 0x7f381c078b20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f383c159380 0x7f383c072420 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f383c0cc8e0 0x7f383c1f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 --2- 172.21.15.163:0/2990535185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f383c06f350 0x7f383c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.071+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 >> 172.21.15.163:0/2990535185 conn(0x7f383c06d420 msgr2=0x7f383c0b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.072+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 shutdown_connections 2024-07-26T06:17:30.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.072+0000 7f3841600640 1 -- 172.21.15.163:0/2990535185 wait complete. 2024-07-26T06:17:30.092 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:17:30.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:30.093 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:30.194 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:17:30.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:30.195 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:17:30.244 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:17:30.244 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:30.245 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:30.302 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:17:30.303 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:17:30.303 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:30.303 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:30.359 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:17:30.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:30.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.570+0000 7f926e200640 1 -- 172.21.15.163:0/385860641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92680b2f90 msgr2=0x7f9268159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.570+0000 7f926e200640 1 --2- 172.21.15.163:0/385860641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92680b2f90 0x7f9268159380 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f92680b47b0 tx=0x7f925002a4a0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.571+0000 7f926e200640 1 -- 172.21.15.163:0/385860641 shutdown_connections 2024-07-26T06:17:30.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.571+0000 7f926e200640 1 --2- 172.21.15.163:0/385860641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92680b2f90 0x7f9268159380 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.571+0000 7f926e200640 1 --2- 172.21.15.163:0/385860641 >> v2:172.21.15.164:3300/0 conn(0x7f926806bf40 0x7f926806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.571+0000 7f926e200640 1 -- 172.21.15.163:0/385860641 >> 172.21.15.163:0/385860641 conn(0x7f926806d420 msgr2=0x7f926806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:30.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.571+0000 7f926e200640 1 -- 172.21.15.163:0/385860641 shutdown_connections 2024-07-26T06:17:30.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.571+0000 7f926e200640 1 -- 172.21.15.163:0/385860641 wait complete. 2024-07-26T06:17:30.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.572+0000 7f926e200640 1 Processor -- start 2024-07-26T06:17:30.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.573+0000 7f926e200640 1 -- start start 2024-07-26T06:17:30.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.573+0000 7f926e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f926806bf40 0x7f92680b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:30.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.573+0000 7f926e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 0x7f926814ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:30.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f926e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92680b2f90 0x7f92680ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:30.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f926e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9268164280 con 0x7f926806f350 2024-07-26T06:17:30.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f926e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f92680cdc10 con 0x7f926806bf40 2024-07-26T06:17:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f926e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9268164400 con 0x7f92680b2f90 2024-07-26T06:17:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 0x7f926814ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 0x7f926814ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50452/0 (socket says 172.21.15.163:50452) 2024-07-26T06:17:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 -- 172.21.15.163:0/926757562 learned_addr learned my addr 172.21.15.163:0/926757562 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f9267400640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f926806bf40 0x7f92680b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:30.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f9267e00640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92680b2f90 0x7f92680ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:30.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 -- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92680b2f90 msgr2=0x7f92680ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92680b2f90 0x7f92680ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 -- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f926806bf40 msgr2=0x7f92680b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f926806bf40 0x7f92680b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.574+0000 7f925ea00640 1 -- 172.21.15.163:0/926757562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926814d6c0 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.575+0000 7f925ea00640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 0x7f926814ecd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9250009920 tx=0x7f9250033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.575+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250031d60 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.575+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9250033e10 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.575+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926814d9a0 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.575+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250005260 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.575+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926814df60 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.576+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926806c3b0 con 0x7f926806f350 2024-07-26T06:17:30.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.577+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f9250005400 con 0x7f926806f350 2024-07-26T06:17:30.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.578+0000 7f925fe00640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9244076630 0x7f9244078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:30.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.579+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f92500b8fa0 con 0x7f926806f350 2024-07-26T06:17:30.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.579+0000 7f9267400640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9244076630 0x7f9244078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:30.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.580+0000 7f9267400640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9244076630 0x7f9244078af0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9258001700 tx=0x7f9258009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:30.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.583+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f925003f030 con 0x7f926806f350 2024-07-26T06:17:30.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.826+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9268072060 con 0x7f926806f350 2024-07-26T06:17:30.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.828+0000 7f925fe00640 1 -- 172.21.15.163:0/926757562 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5232 (secure 0 0 0) 0x7f92500822b0 con 0x7f926806f350 2024-07-26T06:17:30.829 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:30.829 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":36,"btime":"2024-07-26T06:17:14:361523+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:02.717555+0000","modified":"2024-07-26T06:17:05.791542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/3071317413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3071317413},{"type":"v1","addr":"172.21.15.164:6835","nonce":3071317413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5053,"qdb_cluster":[5053]},"id":4}]} 2024-07-26T06:17:30.829 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 36 2024-07-26T06:17:30.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.830+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9244076630 msgr2=0x7f9244078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.830+0000 7f926e200640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9244076630 0x7f9244078af0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9258001700 tx=0x7f9258009040 comp rx=0 tx=0).stop 2024-07-26T06:17:30.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.830+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 msgr2=0x7f926814ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:30.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.830+0000 7f926e200640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 0x7f926814ecd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9250009920 tx=0x7f9250033c20 comp rx=0 tx=0).stop 2024-07-26T06:17:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 shutdown_connections 2024-07-26T06:17:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9244076630 0x7f9244078af0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92680b2f90 0x7f92680ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926806f350 0x7f926814ecd0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 --2- 172.21.15.163:0/926757562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f926806bf40 0x7f92680b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 >> 172.21.15.163:0/926757562 conn(0x7f926806d420 msgr2=0x7f926806d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:30.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 shutdown_connections 2024-07-26T06:17:30.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:30.831+0000 7f926e200640 1 -- 172.21.15.163:0/926757562 wait complete. 2024-07-26T06:17:30.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:31.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.050+0000 7f238fe00640 1 -- 172.21.15.163:0/1844708930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23901556f0 msgr2=0x7f23900cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.050+0000 7f238fe00640 1 --2- 172.21.15.163:0/1844708930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23901556f0 0x7f23900cdd90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2378009830 tx=0x7f237802a510 comp rx=0 tx=0).stop 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.050+0000 7f238fe00640 1 -- 172.21.15.163:0/1844708930 shutdown_connections 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.050+0000 7f238fe00640 1 --2- 172.21.15.163:0/1844708930 >> v2:172.21.15.164:3300/0 conn(0x7f23900ce2d0 0x7f2390071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.050+0000 7f238fe00640 1 --2- 172.21.15.163:0/1844708930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23901556f0 0x7f23900cdd90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.050+0000 7f238fe00640 1 -- 172.21.15.163:0/1844708930 >> 172.21.15.163:0/1844708930 conn(0x7f23900b0e30 msgr2=0x7f23900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.051+0000 7f238fe00640 1 -- 172.21.15.163:0/1844708930 shutdown_connections 2024-07-26T06:17:31.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.051+0000 7f238fe00640 1 -- 172.21.15.163:0/1844708930 wait complete. 2024-07-26T06:17:31.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.052+0000 7f238fe00640 1 Processor -- start 2024-07-26T06:17:31.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.052+0000 7f238fe00640 1 -- start start 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.053+0000 7f238fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2390071a70 0x7f23900a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.053+0000 7f238fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23900ce2d0 0x7f239014cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.053+0000 7f238fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 0x7f239014d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.053+0000 7f238fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f239015f310 con 0x7f23901556f0 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.053+0000 7f238fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2390071ee0 con 0x7f2390071a70 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.053+0000 7f238fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f239015f490 con 0x7f23900ce2d0 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 0x7f239014d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 0x7f239014d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50460/0 (socket says 172.21.15.163:50460) 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 -- 172.21.15.163:0/2307739974 learned_addr learned my addr 172.21.15.163:0/2307739974 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238ea00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2390071a70 0x7f23900a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238e000640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23900ce2d0 0x7f239014cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 -- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23900ce2d0 msgr2=0x7f239014cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23900ce2d0 0x7f239014cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 -- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2390071a70 msgr2=0x7f23900a9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2390071a70 0x7f23900a9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 -- 172.21.15.163:0/2307739974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23900aa2b0 con 0x7f23901556f0 2024-07-26T06:17:31.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.054+0000 7f238f400640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 0x7f239014d0b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f238800c430 tx=0x7f238800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:31.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.055+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388017070 con 0x7f23901556f0 2024-07-26T06:17:31.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.055+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23880053c0 con 0x7f23901556f0 2024-07-26T06:17:31.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.055+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23900b05d0 con 0x7f23901556f0 2024-07-26T06:17:31.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.055+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388003850 con 0x7f23901556f0 2024-07-26T06:17:31.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.055+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23901f1c50 con 0x7f23901556f0 2024-07-26T06:17:31.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.056+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2390071530 con 0x7f23901556f0 2024-07-26T06:17:31.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.057+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f238801a070 con 0x7f23901556f0 2024-07-26T06:17:31.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.058+0000 7f2386a00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f236c076630 0x7f236c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.058+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f2388097170 con 0x7f23901556f0 2024-07-26T06:17:31.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.059+0000 7f238ea00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f236c076630 0x7f236c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.060+0000 7f238ea00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f236c076630 0x7f236c078af0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2378034000 tx=0x7f2378004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:31.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.063+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2388060680 con 0x7f23901556f0 2024-07-26T06:17:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.313+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23900b2870 con 0x7f23901556f0 2024-07-26T06:17:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.314+0000 7f2386a00640 1 -- 172.21.15.163:0/2307739974 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5232 (secure 0 0 0) 0x7f238805ff80 con 0x7f23901556f0 2024-07-26T06:17:31.315 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:31.315 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":36,"btime":"2024-07-26T06:17:14:361523+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:02.717555+0000","modified":"2024-07-26T06:17:05.791542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/3071317413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3071317413},{"type":"v1","addr":"172.21.15.164:6835","nonce":3071317413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5053,"qdb_cluster":[5053]},"id":4}]} 2024-07-26T06:17:31.315 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 36 2024-07-26T06:17:31.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f236c076630 msgr2=0x7f236c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f236c076630 0x7f236c078af0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2378034000 tx=0x7f2378004730 comp rx=0 tx=0).stop 2024-07-26T06:17:31.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 msgr2=0x7f239014d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 0x7f239014d0b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f238800c430 tx=0x7f238800c8f0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 shutdown_connections 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f236c076630 0x7f236c078af0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23901556f0 0x7f239014d0b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23900ce2d0 0x7f239014cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 --2- 172.21.15.163:0/2307739974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2390071a70 0x7f23900a9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.316+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 >> 172.21.15.163:0/2307739974 conn(0x7f23900b0e30 msgr2=0x7f23900b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.317+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 shutdown_connections 2024-07-26T06:17:31.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.317+0000 7f238fe00640 1 -- 172.21.15.163:0/2307739974 wait complete. 2024-07-26T06:17:31.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.536+0000 7f7944a00640 1 -- 172.21.15.163:0/1966357178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7924004800 msgr2=0x7f7924004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.536+0000 7f7944a00640 1 --2- 172.21.15.163:0/1966357178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7924004800 0x7f7924004c80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f792800c2e0 tx=0x7f7924005830 comp rx=0 tx=0).stop 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.537+0000 7f7944a00640 1 -- 172.21.15.163:0/1966357178 shutdown_connections 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.537+0000 7f7944a00640 1 --2- 172.21.15.163:0/1966357178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7924004800 0x7f7924004c80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.537+0000 7f7944a00640 1 -- 172.21.15.163:0/1966357178 >> 172.21.15.163:0/1966357178 conn(0x7f79400b0e30 msgr2=0x7f79400b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.537+0000 7f7944a00640 1 -- 172.21.15.163:0/1966357178 shutdown_connections 2024-07-26T06:17:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.537+0000 7f7944a00640 1 -- 172.21.15.163:0/1966357178 wait complete. 2024-07-26T06:17:31.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.538+0000 7f7944a00640 1 Processor -- start 2024-07-26T06:17:31.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.538+0000 7f7944a00640 1 -- start start 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.539+0000 7f7944a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7924004800 0x7f79400afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.539+0000 7f7944a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 0x7f7940158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.539+0000 7f7944a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79400b2690 0x7f79400b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.539+0000 7f7944a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f79400ce120 con 0x7f79400aace0 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.539+0000 7f7944a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f79400cdfa0 con 0x7f79400b2690 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.539+0000 7f7944a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f79400736c0 con 0x7f7924004800 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 0x7f7940158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 0x7f7940158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50468/0 (socket says 172.21.15.163:50468) 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 -- 172.21.15.163:0/3172162095 learned_addr learned my addr 172.21.15.163:0/3172162095 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:31.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793f400640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7924004800 0x7f79400afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793fe00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79400b2690 0x7f79400b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 -- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7924004800 msgr2=0x7f79400afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7924004800 0x7f79400afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 -- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79400b2690 msgr2=0x7f79400b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79400b2690 0x7f79400b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 -- 172.21.15.163:0/3172162095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79400a9c40 con 0x7f79400aace0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793f400640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7924004800 0x7f79400afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793fe00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79400b2690 0x7f79400b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.540+0000 7f793ea00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 0x7f7940158660 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f792800c400 tx=0x7f7928001d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.541+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928031040 con 0x7f79400aace0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.541+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79280035f0 con 0x7f79400aace0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.541+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928003a60 con 0x7f79400aace0 2024-07-26T06:17:31.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.541+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79400a9f20 con 0x7f79400aace0 2024-07-26T06:17:31.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.541+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79400aa4e0 con 0x7f79400aace0 2024-07-26T06:17:31.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.542+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79400718a0 con 0x7f79400aace0 2024-07-26T06:17:31.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.543+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f7928034070 con 0x7f79400aace0 2024-07-26T06:17:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.544+0000 7f7937400640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f791c076660 0x7f791c078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.544+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f79280b7cf0 con 0x7f79400aace0 2024-07-26T06:17:31.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.545+0000 7f793f400640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f791c076660 0x7f791c078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.546+0000 7f793f400640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f791c076660 0x7f791c078b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f794006b090 tx=0x7f79300093c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:31.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.549+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f79280810f0 con 0x7f79400aace0 2024-07-26T06:17:31.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.756+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79400b34f0 con 0x7f79400aace0 2024-07-26T06:17:31.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.758+0000 7f7937400640 1 -- 172.21.15.163:0/3172162095 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16689 (secure 0 0 0) 0x7f79280809f0 con 0x7f79400aace0 2024-07-26T06:17:31.759 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:31.759 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":58,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:04.288135+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:00.658125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:01.704071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f791c076660 msgr2=0x7f791c078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f791c076660 0x7f791c078b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f794006b090 tx=0x7f79300093c0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 msgr2=0x7f7940158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 0x7f7940158660 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f792800c400 tx=0x7f7928001d10 comp rx=0 tx=0).stop 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 shutdown_connections 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f791c076660 0x7f791c078b20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79400b2690 0x7f79400b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f79400aace0 0x7f7940158660 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 --2- 172.21.15.163:0/3172162095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7924004800 0x7f79400afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 >> 172.21.15.163:0/3172162095 conn(0x7f79400b0e30 msgr2=0x7f7940151d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 shutdown_connections 2024-07-26T06:17:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.761+0000 7f7944a00640 1 -- 172.21.15.163:0/3172162095 wait complete. 2024-07-26T06:17:31.782 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:17:31.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 -- 172.21.15.163:0/2794410199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 msgr2=0x7f636c06c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 --2- 172.21.15.163:0/2794410199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c06c320 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f636c06f660 tx=0x7f636402a8f0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 -- 172.21.15.163:0/2794410199 shutdown_connections 2024-07-26T06:17:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 --2- 172.21.15.163:0/2794410199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c06c320 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 --2- 172.21.15.163:0/2794410199 >> v2:172.21.15.164:3300/0 conn(0x7f636c071cf0 0x7f636c15ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 -- 172.21.15.163:0/2794410199 >> 172.21.15.163:0/2794410199 conn(0x7f636c06d420 msgr2=0x7f636c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 -- 172.21.15.163:0/2794410199 shutdown_connections 2024-07-26T06:17:31.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.989+0000 7f6371800640 1 -- 172.21.15.163:0/2794410199 wait complete. 2024-07-26T06:17:31.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.991+0000 7f6371800640 1 Processor -- start 2024-07-26T06:17:31.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.991+0000 7f6371800640 1 -- start start 2024-07-26T06:17:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f6371800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c1f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f6371800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f636c071cf0 0x7f636c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f6371800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f636c0cb7e0 0x7f636c0ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f6371800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f636c0736b0 con 0x7f636c06bf40 2024-07-26T06:17:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f6371800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f636c073530 con 0x7f636c071cf0 2024-07-26T06:17:31.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f6371800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f636c073830 con 0x7f636c0cb7e0 2024-07-26T06:17:31.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.992+0000 7f636aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c1f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c1f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50488/0 (socket says 172.21.15.163:50488) 2024-07-26T06:17:31.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 -- 172.21.15.163:0/3159473854 learned_addr learned my addr 172.21.15.163:0/3159473854 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:31.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 -- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f636c0cb7e0 msgr2=0x7f636c0ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f636c0cb7e0 0x7f636c0ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 -- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f636c071cf0 msgr2=0x7f636c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:31.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f636c071cf0 0x7f636c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:31.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 -- 172.21.15.163:0/3159473854 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636c0ccbc0 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f636aa00640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c1f35c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b420 tx=0x7f635c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c011830 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f635c016070 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c011d60 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.993+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636c0ccea0 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.994+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f636c0cd460 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.995+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f636c073d50 con 0x7f636c06bf40 2024-07-26T06:17:31.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.996+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f635c01b460 con 0x7f636c06bf40 2024-07-26T06:17:31.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.997+0000 7f6363400640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6348076630 0x7f6348078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:31.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.998+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f635c097890 con 0x7f636c06bf40 2024-07-26T06:17:31.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.998+0000 7f6362000640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6348076630 0x7f6348078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:31.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:31.999+0000 7f6362000640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6348076630 0x7f6348078af0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6354005fd0 tx=0x7f6354005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:32.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.002+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f635c020020 con 0x7f636c06bf40 2024-07-26T06:17:32.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.232+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f636c155470 con 0x7f636c06bf40 2024-07-26T06:17:32.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.234+0000 7f6363400640 1 -- 172.21.15.163:0/3159473854 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5232 (secure 0 0 0) 0x7f635c060da0 con 0x7f636c06bf40 2024-07-26T06:17:32.235 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:32.235 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":36,"btime":"2024-07-26T06:17:14:361523+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:02.717555+0000","modified":"2024-07-26T06:17:05.791542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/3071317413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3071317413},{"type":"v1","addr":"172.21.15.164:6835","nonce":3071317413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5053,"qdb_cluster":[5053]},"id":4}]} 2024-07-26T06:17:32.235 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 36 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6348076630 msgr2=0x7f6348078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6348076630 0x7f6348078af0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6354005fd0 tx=0x7f6354005850 comp rx=0 tx=0).stop 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 msgr2=0x7f636c1f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c1f35c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b420 tx=0x7f635c00b8e0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 shutdown_connections 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6348076630 0x7f6348078af0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f636c0cb7e0 0x7f636c0ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f636c071cf0 0x7f636c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 --2- 172.21.15.163:0/3159473854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f636c06bf40 0x7f636c1f35c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.236+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 >> 172.21.15.163:0/3159473854 conn(0x7f636c06d420 msgr2=0x7f636c0b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:32.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.237+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 shutdown_connections 2024-07-26T06:17:32.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.237+0000 7f6371800640 1 -- 172.21.15.163:0/3159473854 wait complete. 2024-07-26T06:17:32.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.455+0000 7f6034c00640 1 -- 172.21.15.163:0/1900582776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014003a30 msgr2=0x7f6014003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.455+0000 7f6034c00640 1 --2- 172.21.15.163:0/1900582776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6014003a30 0x7f6014003e50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f602000c260 tx=0x7f6014005830 comp rx=0 tx=0).stop 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.456+0000 7f602fe00640 1 -- 172.21.15.163:0/1900582776 reap_dead start 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.456+0000 7f6034c00640 1 -- 172.21.15.163:0/1900582776 shutdown_connections 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.456+0000 7f6034c00640 1 -- 172.21.15.163:0/1900582776 >> 172.21.15.163:0/1900582776 conn(0x7f60300b0e30 msgr2=0x7f60300b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.456+0000 7f6034c00640 1 -- 172.21.15.163:0/1900582776 shutdown_connections 2024-07-26T06:17:32.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.456+0000 7f6034c00640 1 -- 172.21.15.163:0/1900582776 wait complete. 2024-07-26T06:17:32.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.457+0000 7f6034c00640 1 Processor -- start 2024-07-26T06:17:32.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.458+0000 7f6034c00640 1 -- start start 2024-07-26T06:17:32.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.458+0000 7f6034c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6014003a30 0x7f60300cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.458+0000 7f6034c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60300ad910 0x7f60300add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.458+0000 7f6034c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 0x7f60300cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f6034c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60300718c0 con 0x7f60300cd0a0 2024-07-26T06:17:32.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f6034c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6030071740 con 0x7f6014003a30 2024-07-26T06:17:32.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f6034c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60300b3e20 con 0x7f60300ad910 2024-07-26T06:17:32.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 0x7f60300cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:32.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 0x7f60300cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50496/0 (socket says 172.21.15.163:50496) 2024-07-26T06:17:32.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 -- 172.21.15.163:0/562119834 learned_addr learned my addr 172.21.15.163:0/562119834 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 -- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60300ad910 msgr2=0x7f60300add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60300ad910 0x7f60300add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 -- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6014003a30 msgr2=0x7f60300cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6014003a30 0x7f60300cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 -- 172.21.15.163:0/562119834 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60300cc9b0 con 0x7f60300cd0a0 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.459+0000 7f602fe00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 0x7f60300cd520 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6030068de0 tx=0x7f602800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.460+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602800f040 con 0x7f60300cd0a0 2024-07-26T06:17:32.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.460+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6028018070 con 0x7f60300cd0a0 2024-07-26T06:17:32.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.460+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60301f4190 con 0x7f60300cd0a0 2024-07-26T06:17:32.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.460+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602800c900 con 0x7f60300cd0a0 2024-07-26T06:17:32.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.460+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60301f4710 con 0x7f60300cd0a0 2024-07-26T06:17:32.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.461+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60300b4380 con 0x7f60300cd0a0 2024-07-26T06:17:32.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.462+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f602800caa0 con 0x7f60300cd0a0 2024-07-26T06:17:32.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.463+0000 7f6027400640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6010076630 0x7f6010078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.463+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f6028097970 con 0x7f60300cd0a0 2024-07-26T06:17:32.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.464+0000 7f602f400640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6010076630 0x7f6010078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:32.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.465+0000 7f602f400640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6010076630 0x7f6010078af0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6020033000 tx=0x7f6020000fe0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:32.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.467+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60300cc9b0 con 0x7f60300cd0a0 2024-07-26T06:17:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.712+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60300b34f0 con 0x7f60300cd0a0 2024-07-26T06:17:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.713+0000 7f6027400640 1 -- 172.21.15.163:0/562119834 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5232 (secure 0 0 0) 0x7f6028060d70 con 0x7f60300cd0a0 2024-07-26T06:17:32.715 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:32.715 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":36,"btime":"2024-07-26T06:17:14:361523+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:02.717555+0000","modified":"2024-07-26T06:17:05.791542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6835/3071317413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3071317413},{"type":"v1","addr":"172.21.15.164:6835","nonce":3071317413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5053,"qdb_cluster":[5053]},"id":4}]} 2024-07-26T06:17:32.715 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 36 2024-07-26T06:17:32.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6010076630 msgr2=0x7f6010078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6010076630 0x7f6010078af0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6020033000 tx=0x7f6020000fe0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 msgr2=0x7f60300cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 0x7f60300cd520 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6030068de0 tx=0x7f602800ec20 comp rx=0 tx=0).stop 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 shutdown_connections 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6010076630 0x7f6010078af0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60300cd0a0 0x7f60300cd520 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60300ad910 0x7f60300add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 --2- 172.21.15.163:0/562119834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6014003a30 0x7f60300cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.716+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 >> 172.21.15.163:0/562119834 conn(0x7f60300b0e30 msgr2=0x7f603015bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.717+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 shutdown_connections 2024-07-26T06:17:32.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.717+0000 7f6034c00640 1 -- 172.21.15.163:0/562119834 wait complete. 2024-07-26T06:17:32.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:32.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 -- 172.21.15.163:0/2176135909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 msgr2=0x7f09000b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2176135909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f09000b6110 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f09000b8f20 tx=0x7f08e802a4f0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 -- 172.21.15.163:0/2176135909 shutdown_connections 2024-07-26T06:17:32.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2176135909 >> v2:172.21.15.164:3300/0 conn(0x7f09000b6650 0x7f09000d24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2176135909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f09000b6110 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2176135909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09000cd910 0x7f09000cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 -- 172.21.15.163:0/2176135909 >> 172.21.15.163:0/2176135909 conn(0x7f09000cfd40 msgr2=0x7f09000d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:32.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 -- 172.21.15.163:0/2176135909 shutdown_connections 2024-07-26T06:17:32.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.941+0000 7f08ffe00640 1 -- 172.21.15.163:0/2176135909 wait complete. 2024-07-26T06:17:32.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.943+0000 7f08ffe00640 1 Processor -- start 2024-07-26T06:17:32.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.943+0000 7f08ffe00640 1 -- start start 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ffe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09000b6650 0x7f09001572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ffe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09000cd910 0x7f090015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ffe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f0900156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ffe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f09000b4ec0 con 0x7f09000d0e90 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ffe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f09000b4d40 con 0x7f09000b6650 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ffe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f09000b5040 con 0x7f09000cd910 2024-07-26T06:17:32.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.944+0000 7f08ff400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f0900156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:32.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f0900156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50514/0 (socket says 172.21.15.163:50514) 2024-07-26T06:17:32.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 -- 172.21.15.163:0/2943962260 learned_addr learned my addr 172.21.15.163:0/2943962260 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08fea00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09000b6650 0x7f09001572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08f6000640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09000cd910 0x7f090015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 -- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09000cd910 msgr2=0x7f090015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09000cd910 0x7f090015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 -- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09000b6650 msgr2=0x7f09001572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09000b6650 0x7f09001572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 -- 172.21.15.163:0/2943962260 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09001570f0 con 0x7f09000d0e90 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.945+0000 7f08ff400640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f0900156bb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f08f800be50 tx=0x7f08f800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:32.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.946+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8019070 con 0x7f09000d0e90 2024-07-26T06:17:32.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.946+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08f8014440 con 0x7f09000d0e90 2024-07-26T06:17:32.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.946+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8014890 con 0x7f09000d0e90 2024-07-26T06:17:32.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.946+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09001579c0 con 0x7f09000d0e90 2024-07-26T06:17:32.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.946+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0900157f80 con 0x7f09000d0e90 2024-07-26T06:17:32.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.947+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09000d2b50 con 0x7f09000d0e90 2024-07-26T06:17:32.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.948+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f08f801c070 con 0x7f09000d0e90 2024-07-26T06:17:32.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.949+0000 7f08f7400640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f08dc076630 0x7f08dc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:32.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.949+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f08f8097dc0 con 0x7f09000d0e90 2024-07-26T06:17:32.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.950+0000 7f08fea00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f08dc076630 0x7f08dc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:32.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.951+0000 7f08fea00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f08dc076630 0x7f08dc078af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009890 tx=0x7f08f0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:32.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:32.953+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08f80612d0 con 0x7f09000d0e90 2024-07-26T06:17:33.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.167+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f090015fa30 con 0x7f09000d0e90 2024-07-26T06:17:33.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.169+0000 7f08f7400640 1 -- 172.21.15.163:0/2943962260 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16689 (secure 0 0 0) 0x7f08f8060bd0 con 0x7f09000d0e90 2024-07-26T06:17:33.170 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:33.170 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":58,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:04.288135+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:00.658125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:01.704071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f08dc076630 msgr2=0x7f08dc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f08dc076630 0x7f08dc078af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009890 tx=0x7f08f0008040 comp rx=0 tx=0).stop 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 msgr2=0x7f0900156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f0900156bb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f08f800be50 tx=0x7f08f800bf40 comp rx=0 tx=0).stop 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 shutdown_connections 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f08dc076630 0x7f08dc078af0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09000d0e90 0x7f0900156bb0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09000cd910 0x7f090015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 --2- 172.21.15.163:0/2943962260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09000b6650 0x7f09001572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 >> 172.21.15.163:0/2943962260 conn(0x7f09000cfd40 msgr2=0x7f09000d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:33.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 shutdown_connections 2024-07-26T06:17:33.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.172+0000 7f08ffe00640 1 -- 172.21.15.163:0/2943962260 wait complete. 2024-07-26T06:17:33.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 -- 172.21.15.163:0/1206976693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a40ce4a0 msgr2=0x7f35a4071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 --2- 172.21.15.163:0/1206976693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a40ce4a0 0x7f35a4071530 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f358c009830 tx=0x7f358c02a510 comp rx=0 tx=0).stop 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 -- 172.21.15.163:0/1206976693 shutdown_connections 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 --2- 172.21.15.163:0/1206976693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a4071a70 0x7f35a40b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 --2- 172.21.15.163:0/1206976693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a40ce4a0 0x7f35a4071530 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 -- 172.21.15.163:0/1206976693 >> 172.21.15.163:0/1206976693 conn(0x7f35a40b0e30 msgr2=0x7f35a40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 -- 172.21.15.163:0/1206976693 shutdown_connections 2024-07-26T06:17:33.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.396+0000 7f35a3e00640 1 -- 172.21.15.163:0/1206976693 wait complete. 2024-07-26T06:17:33.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.398+0000 7f35a3e00640 1 Processor -- start 2024-07-26T06:17:33.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.398+0000 7f35a3e00640 1 -- start start 2024-07-26T06:17:33.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 0x7f35a40b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a40ce4a0 0x7f35a41f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a3e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a41556f0 0x7f35a414e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a3e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35a40b3fe0 con 0x7f35a4071a70 2024-07-26T06:17:33.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a3e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f35a40b3e60 con 0x7f35a41556f0 2024-07-26T06:17:33.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a3e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35a40b4160 con 0x7f35a40ce4a0 2024-07-26T06:17:33.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.399+0000 7f35a2a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 0x7f35a40b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:33.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 0x7f35a40b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50536/0 (socket says 172.21.15.163:50536) 2024-07-26T06:17:33.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 -- 172.21.15.163:0/3340584226 learned_addr learned my addr 172.21.15.163:0/3340584226 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 -- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a40ce4a0 msgr2=0x7f35a41f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a40ce4a0 0x7f35a41f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 -- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a41556f0 msgr2=0x7f35a414e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a41556f0 0x7f35a414e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 -- 172.21.15.163:0/3340584226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a41f47a0 con 0x7f35a4071a70 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f35a2a00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 0x7f35a40b0250 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f359400c3a0 tx=0x7f359400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.400+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594011790 con 0x7f35a4071a70 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.401+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3594017070 con 0x7f35a4071a70 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.401+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594011ce0 con 0x7f35a4071a70 2024-07-26T06:17:33.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.401+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a41f4a80 con 0x7f35a4071a70 2024-07-26T06:17:33.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.401+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a414e150 con 0x7f35a4071a70 2024-07-26T06:17:33.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.402+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a40b46f0 con 0x7f35a4071a70 2024-07-26T06:17:33.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.403+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f359401c460 con 0x7f35a4071a70 2024-07-26T06:17:33.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.404+0000 7f359aa00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3580076630 0x7f3580078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.404+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f3594097ac0 con 0x7f35a4071a70 2024-07-26T06:17:33.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.405+0000 7f359be00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3580076630 0x7f3580078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:33.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.406+0000 7f359be00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3580076630 0x7f3580078af0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f358c034000 tx=0x7f358c0046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:33.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.408+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3594061060 con 0x7f35a4071a70 2024-07-26T06:17:33.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.656+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f35a40b34f0 con 0x7f35a4071a70 2024-07-26T06:17:33.658 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:17:33.658+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:17:33.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.661+0000 7f359aa00640 1 -- 172.21.15.163:0/3340584226 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7f3594060960 con 0x7f35a4071a70 2024-07-26T06:17:33.662 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:17:33.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3580076630 msgr2=0x7f3580078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3580076630 0x7f3580078af0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f358c034000 tx=0x7f358c0046c0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 msgr2=0x7f35a40b0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 0x7f35a40b0250 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f359400c3a0 tx=0x7f359400c860 comp rx=0 tx=0).stop 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 shutdown_connections 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3580076630 0x7f3580078af0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a41556f0 0x7f35a414e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a40ce4a0 0x7f35a41f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 --2- 172.21.15.163:0/3340584226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35a4071a70 0x7f35a40b0250 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.664+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 >> 172.21.15.163:0/3340584226 conn(0x7f35a40b0e30 msgr2=0x7f35a4151d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.665+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 shutdown_connections 2024-07-26T06:17:33.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.665+0000 7f35a3e00640 1 -- 172.21.15.163:0/3340584226 wait complete. 2024-07-26T06:17:33.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:17:33.788 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 -- 172.21.15.163:0/3222249209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc15ccb0 msgr2=0x7f94fc15d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 --2- 172.21.15.163:0/3222249209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc15ccb0 0x7f94fc15d160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f94fc075b80 tx=0x7f94f402aa10 comp rx=0 tx=0).stop 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 -- 172.21.15.163:0/3222249209 shutdown_connections 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 --2- 172.21.15.163:0/3222249209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc15ccb0 0x7f94fc15d160 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 --2- 172.21.15.163:0/3222249209 >> v2:172.21.15.164:3300/0 conn(0x7f94fc06f2f0 0x7f94fc0b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 --2- 172.21.15.163:0/3222249209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94fc0ccdc0 0x7f94fc1590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.874+0000 7f9501c00640 1 -- 172.21.15.163:0/3222249209 >> 172.21.15.163:0/3222249209 conn(0x7f94fc0b0e30 msgr2=0x7f94fc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:33.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.875+0000 7f9501c00640 1 -- 172.21.15.163:0/3222249209 shutdown_connections 2024-07-26T06:17:33.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.875+0000 7f9501c00640 1 -- 172.21.15.163:0/3222249209 wait complete. 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.876+0000 7f9501c00640 1 Processor -- start 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.876+0000 7f9501c00640 1 -- start start 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f9501c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94fc06f2f0 0x7f94fc142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f9501c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 0x7f94fc154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f9501c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94fc15ccb0 0x7f94fc141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f9501c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc15d820 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f9501c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc15d6a0 con 0x7f94fc06f2f0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f9501c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc15d9a0 con 0x7f94fc15ccb0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f94fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 0x7f94fc154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f94fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 0x7f94fc154330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50546/0 (socket says 172.21.15.163:50546) 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.877+0000 7f94fb400640 1 -- 172.21.15.163:0/3108434095 learned_addr learned my addr 172.21.15.163:0/3108434095 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94fb400640 1 -- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94fc15ccb0 msgr2=0x7f94fc141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94fb400640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94fc15ccb0 0x7f94fc141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94fb400640 1 -- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94fc06f2f0 msgr2=0x7f94fc142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94fb400640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94fc06f2f0 0x7f94fc142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94fb400640 1 -- 172.21.15.163:0/3108434095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc14f6a0 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94fb400640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 0x7f94fc154330 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f94e800c970 tx=0x7f94e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e8007c10 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94e8018480 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e80188d0 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.878+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94fc14f940 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.879+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94fc15b0b0 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.880+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94fc1590a0 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.881+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f94e8018a70 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.882+0000 7f94f3e00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f94dc076630 0x7f94dc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:33.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.882+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f94e80989b0 con 0x7f94fc0ccdc0 2024-07-26T06:17:33.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.882+0000 7f94fbe00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f94dc076630 0x7f94dc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:33.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.883+0000 7f94fbe00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f94dc076630 0x7f94dc078af0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f94fc154d70 tx=0x7f94ec005b60 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:33.886+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94e8061ce0 con 0x7f94fc0ccdc0 2024-07-26T06:17:34.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.134+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f94fc156830 con 0x7f94fc0ccdc0 2024-07-26T06:17:34.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.672+0000 7f94f3e00640 1 -- 172.21.15.163:0/3108434095 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7f94e80615e0 con 0x7f94fc0ccdc0 2024-07-26T06:17:34.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.675+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f94dc076630 msgr2=0x7f94dc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:34.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.675+0000 7f9501c00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f94dc076630 0x7f94dc078af0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f94fc154d70 tx=0x7f94ec005b60 comp rx=0 tx=0).stop 2024-07-26T06:17:34.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.675+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 msgr2=0x7f94fc154330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.675+0000 7f9501c00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 0x7f94fc154330 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f94e800c970 tx=0x7f94e800ce30 comp rx=0 tx=0).stop 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 shutdown_connections 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f94dc076630 0x7f94dc078af0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94fc15ccb0 0x7f94fc141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94fc0ccdc0 0x7f94fc154330 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 --2- 172.21.15.163:0/3108434095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94fc06f2f0 0x7f94fc142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 >> 172.21.15.163:0/3108434095 conn(0x7f94fc0b0e30 msgr2=0x7f94fc157310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:34.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 shutdown_connections 2024-07-26T06:17:34.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.676+0000 7f9501c00640 1 -- 172.21.15.163:0/3108434095 wait complete. 2024-07-26T06:17:34.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:17:34.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 -- 172.21.15.163:0/27912529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca8071c40 msgr2=0x7f2ca80b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:34.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 --2- 172.21.15.163:0/27912529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca8071c40 0x7f2ca80b3710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005fd0 tx=0x7f2ca002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 -- 172.21.15.163:0/27912529 shutdown_connections 2024-07-26T06:17:34.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 --2- 172.21.15.163:0/27912529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca8071c40 0x7f2ca80b3710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 --2- 172.21.15.163:0/27912529 >> v2:172.21.15.164:3300/0 conn(0x7f2ca81556f0 0x7f2ca80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 -- 172.21.15.163:0/27912529 >> 172.21.15.163:0/27912529 conn(0x7f2ca80b0e30 msgr2=0x7f2ca80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:34.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 -- 172.21.15.163:0/27912529 shutdown_connections 2024-07-26T06:17:34.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.903+0000 7f2cade00640 1 -- 172.21.15.163:0/27912529 wait complete. 2024-07-26T06:17:34.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.905+0000 7f2cade00640 1 Processor -- start 2024-07-26T06:17:34.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.905+0000 7f2cade00640 1 -- start start 2024-07-26T06:17:34.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.906+0000 7f2cade00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca8071c40 0x7f2ca813f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:34.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cade00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca80ce2d0 0x7f2ca815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cade00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 0x7f2ca815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cade00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca80b3fe0 con 0x7f2ca81556f0 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cade00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca80b3e60 con 0x7f2ca8071c40 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cade00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca80b4160 con 0x7f2ca80ce2d0 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 0x7f2ca815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 0x7f2ca815d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48684/0 (socket says 172.21.15.163:48684) 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 -- 172.21.15.163:0/3629884317 learned_addr learned my addr 172.21.15.163:0/3629884317 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2caca00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca8071c40 0x7f2ca813f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2c9fe00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca80ce2d0 0x7f2ca815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:34.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 -- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca80ce2d0 msgr2=0x7f2ca815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca80ce2d0 0x7f2ca815cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 -- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca8071c40 msgr2=0x7f2ca813f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.907+0000 7f2cad400640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca8071c40 0x7f2ca813f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2cad400640 1 -- 172.21.15.163:0/3629884317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca813fec0 con 0x7f2ca81556f0 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2cad400640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 0x7f2ca815d220 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009cd0 tx=0x7f2ca0031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0040070 con 0x7f2ca81556f0 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ca0004040 con 0x7f2ca81556f0 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca81401a0 con 0x7f2ca81556f0 2024-07-26T06:17:34.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0004490 con 0x7f2ca81556f0 2024-07-26T06:17:34.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.908+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca813d880 con 0x7f2ca81556f0 2024-07-26T06:17:34.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.909+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca813da80 con 0x7f2ca81556f0 2024-07-26T06:17:34.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.911+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f2ca0039040 con 0x7f2ca81556f0 2024-07-26T06:17:34.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.912+0000 7f2ca5600640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c84076630 0x7f2c84078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:34.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.912+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f2ca00b92c0 con 0x7f2ca81556f0 2024-07-26T06:17:34.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.912+0000 7f2caca00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c84076630 0x7f2c84078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:34.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.913+0000 7f2caca00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c84076630 0x7f2c84078af0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2ca814a530 tx=0x7f2c98005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:34.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:34.916+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ca0082050 con 0x7f2ca81556f0 2024-07-26T06:17:35.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.122+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2ca813d390 con 0x7f2ca81556f0 2024-07-26T06:17:35.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.676+0000 7f2ca5600640 1 -- 172.21.15.163:0/3629884317 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) ==== 181+0+0 (secure 0 0 0) 0x7f2ca0086350 con 0x7f2ca81556f0 2024-07-26T06:17:35.677 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:17:35.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c84076630 msgr2=0x7f2c84078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:35.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c84076630 0x7f2c84078af0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2ca814a530 tx=0x7f2c98005f90 comp rx=0 tx=0).stop 2024-07-26T06:17:35.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 msgr2=0x7f2ca815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:35.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 0x7f2ca815d220 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009cd0 tx=0x7f2ca0031700 comp rx=0 tx=0).stop 2024-07-26T06:17:35.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 shutdown_connections 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c84076630 0x7f2c84078af0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ca81556f0 0x7f2ca815d220 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ca80ce2d0 0x7f2ca815cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 --2- 172.21.15.163:0/3629884317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca8071c40 0x7f2ca813f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 >> 172.21.15.163:0/3629884317 conn(0x7f2ca80b0e30 msgr2=0x7f2ca815a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 shutdown_connections 2024-07-26T06:17:35.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.680+0000 7f2cade00640 1 -- 172.21.15.163:0/3629884317 wait complete. 2024-07-26T06:17:35.710 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:17:35.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 -- 172.21.15.163:0/3480927097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10440cbad0 msgr2=0x7f10440cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:35.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 --2- 172.21.15.163:0/3480927097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10440cbad0 0x7f10440cd530 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1034005fd0 tx=0x7f103402a530 comp rx=0 tx=0).stop 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 -- 172.21.15.163:0/3480927097 shutdown_connections 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 --2- 172.21.15.163:0/3480927097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 0x7f10440cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 --2- 172.21.15.163:0/3480927097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10440cbad0 0x7f10440cd530 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 -- 172.21.15.163:0/3480927097 >> 172.21.15.163:0/3480927097 conn(0x7f10440b0e30 msgr2=0x7f10440b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 -- 172.21.15.163:0/3480927097 shutdown_connections 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.909+0000 7f1049400640 1 -- 172.21.15.163:0/3480927097 wait complete. 2024-07-26T06:17:35.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.911+0000 7f1049400640 1 Processor -- start 2024-07-26T06:17:35.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.911+0000 7f1049400640 1 -- start start 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1049400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 0x7f10440c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1049400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10440cd000 0x7f1044157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1049400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 0x7f1044073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1049400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1044159700 con 0x7f1044153e60 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1049400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1044159580 con 0x7f10440cd000 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1049400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1044159880 con 0x7f10440b4420 2024-07-26T06:17:35.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1048a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 0x7f1044073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:35.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1048a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 0x7f1044073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48710/0 (socket says 172.21.15.163:48710) 2024-07-26T06:17:35.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.912+0000 7f1048a00640 1 -- 172.21.15.163:0/2860505877 learned_addr learned my addr 172.21.15.163:0/2860505877 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1043e00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 0x7f10440c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f103b400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10440cd000 0x7f1044157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1048a00640 1 -- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 msgr2=0x7f10440c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1048a00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 0x7f10440c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1048a00640 1 -- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10440cd000 msgr2=0x7f1044157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1048a00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10440cd000 0x7f1044157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1048a00640 1 -- 172.21.15.163:0/2860505877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10440c6010 con 0x7f1044153e60 2024-07-26T06:17:35.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f103b400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10440cd000 0x7f1044157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1043e00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 0x7f10440c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1048a00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 0x7f1044073870 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f103c00be60 tx=0x7f103c00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.913+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c019070 con 0x7f1044153e60 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.914+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f103c014440 con 0x7f1044153e60 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.914+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c0148d0 con 0x7f1044153e60 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.914+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10440c62f0 con 0x7f1044153e60 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.914+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10440c68b0 con 0x7f1044153e60 2024-07-26T06:17:35.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.915+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104415ce50 con 0x7f1044153e60 2024-07-26T06:17:35.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.916+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f103c01c070 con 0x7f1044153e60 2024-07-26T06:17:35.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.917+0000 7f1040c00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1020076660 0x7f1020078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:35.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.917+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5357+0+0 (secure 0 0 0) 0x7f103c097c90 con 0x7f1044153e60 2024-07-26T06:17:35.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.918+0000 7f1043e00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1020076660 0x7f1020078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:35.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.919+0000 7f1043e00640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1020076660 0x7f1020078b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1034033000 tx=0x7f1034004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:35.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:35.921+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f103c061200 con 0x7f1044153e60 2024-07-26T06:17:36.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.130+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f10441e9410 con 0x7f1044153e60 2024-07-26T06:17:36.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.687+0000 7f1040c00640 1 -- 172.21.15.163:0/2860505877 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) ==== 169+0+0 (secure 0 0 0) 0x7f103c060b00 con 0x7f1044153e60 2024-07-26T06:17:36.688 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:17:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.691+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1020076660 msgr2=0x7f1020078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.691+0000 7f1049400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1020076660 0x7f1020078b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1034033000 tx=0x7f1034004730 comp rx=0 tx=0).stop 2024-07-26T06:17:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.691+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 msgr2=0x7f1044073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.691+0000 7f1049400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 0x7f1044073870 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f103c00be60 tx=0x7f103c00bf50 comp rx=0 tx=0).stop 2024-07-26T06:17:36.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.696+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 shutdown_connections 2024-07-26T06:17:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.696+0000 7f1049400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1020076660 0x7f1020078b20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.696+0000 7f1049400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1044153e60 0x7f1044073870 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.696+0000 7f1049400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10440cd000 0x7f1044157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.696+0000 7f1049400640 1 --2- 172.21.15.163:0/2860505877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10440b4420 0x7f10440c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.696+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 >> 172.21.15.163:0/2860505877 conn(0x7f10440b0e30 msgr2=0x7f104415aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.697+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 shutdown_connections 2024-07-26T06:17:36.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.698+0000 7f1049400640 1 -- 172.21.15.163:0/2860505877 wait complete. 2024-07-26T06:17:36.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-07-26T06:17:36.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.914+0000 7f5130800640 1 -- 172.21.15.163:0/3017913486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f512806bf40 msgr2=0x7f512806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:36.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.914+0000 7f5130800640 1 --2- 172.21.15.163:0/3017913486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f512806bf40 0x7f512806c370 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5124005fd0 tx=0x7f512402a810 comp rx=0 tx=0).stop 2024-07-26T06:17:36.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.915+0000 7f5130800640 1 -- 172.21.15.163:0/3017913486 shutdown_connections 2024-07-26T06:17:36.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.915+0000 7f5130800640 1 --2- 172.21.15.163:0/3017913486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f512806bf40 0x7f512806c370 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.915+0000 7f5130800640 1 --2- 172.21.15.163:0/3017913486 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51280cb7e0 0x7f512815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.915+0000 7f5130800640 1 -- 172.21.15.163:0/3017913486 >> 172.21.15.163:0/3017913486 conn(0x7f512806d420 msgr2=0x7f512806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:36.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.915+0000 7f5130800640 1 -- 172.21.15.163:0/3017913486 shutdown_connections 2024-07-26T06:17:36.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.915+0000 7f5130800640 1 -- 172.21.15.163:0/3017913486 wait complete. 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.916+0000 7f5130800640 1 Processor -- start 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.917+0000 7f5130800640 1 -- start start 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.917+0000 7f5130800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512806bf40 0x7f51280ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.917+0000 7f5130800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5128071cf0 0x7f51281f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.917+0000 7f5130800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 0x7f51280b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f5130800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51280736b0 con 0x7f51280cb7e0 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f5130800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5128073530 con 0x7f5128071cf0 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f5130800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5128073830 con 0x7f512806bf40 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 0x7f51280b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 0x7f51280b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48720/0 (socket says 172.21.15.163:48720) 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 -- 172.21.15.163:0/933200307 learned_addr learned my addr 172.21.15.163:0/933200307 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:36.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 -- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512806bf40 msgr2=0x7f51280ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512806bf40 0x7f51280ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 -- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5128071cf0 msgr2=0x7f51281f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5128071cf0 0x7f51281f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.918+0000 7f512e800640 1 -- 172.21.15.163:0/933200307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51281f77d0 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.919+0000 7f512e800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 0x7f51280b04c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f5124009b30 tx=0x7f51240315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.919+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124041070 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.919+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5124031aa0 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.919+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124037040 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.919+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51281f7a70 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.919+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51281f8030 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.920+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5128073d00 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.921+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f5124044070 con 0x7f51280cb7e0 2024-07-26T06:17:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.922+0000 7f5122000640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5108076630 0x7f5108078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:36.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.922+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f51240b8790 con 0x7f51280cb7e0 2024-07-26T06:17:36.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.923+0000 7f512de00640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5108076630 0x7f5108078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:36.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.924+0000 7f512de00640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5108076630 0x7f5108078af0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51140058c0 tx=0x7f5114005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:36.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:36.926+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5124081dd0 con 0x7f51280cb7e0 2024-07-26T06:17:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.133+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f51280aa520 con 0x7f51280cb7e0 2024-07-26T06:17:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.138+0000 7f5122000640 1 -- 172.21.15.163:0/933200307 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v33) ==== 71+0+0 (secure 0 0 0) 0x7f51240816d0 con 0x7f51280cb7e0 2024-07-26T06:17:37.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5108076630 msgr2=0x7f5108078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5108076630 0x7f5108078af0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51140058c0 tx=0x7f5114005850 comp rx=0 tx=0).stop 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 msgr2=0x7f51280b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 0x7f51280b04c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f5124009b30 tx=0x7f51240315b0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 shutdown_connections 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5108076630 0x7f5108078af0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f51280cb7e0 0x7f51280b04c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5128071cf0 0x7f51281f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 --2- 172.21.15.163:0/933200307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512806bf40 0x7f51280ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.141+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 >> 172.21.15.163:0/933200307 conn(0x7f512806d420 msgr2=0x7f51280af540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.142+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 shutdown_connections 2024-07-26T06:17:37.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.142+0000 7f5130800640 1 -- 172.21.15.163:0/933200307 wait complete. 2024-07-26T06:17:37.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-07-26T06:17:37.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 -- 172.21.15.163:0/645977058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640cbad0 msgr2=0x7f53640cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 --2- 172.21.15.163:0/645977058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640cbad0 0x7f53640cd530 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f53640b1bc0 tx=0x7f535402a540 comp rx=0 tx=0).stop 2024-07-26T06:17:37.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 -- 172.21.15.163:0/645977058 shutdown_connections 2024-07-26T06:17:37.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 --2- 172.21.15.163:0/645977058 >> v2:172.21.15.164:3300/0 conn(0x7f53640b4420 0x7f53640cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 --2- 172.21.15.163:0/645977058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640cbad0 0x7f53640cd530 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 -- 172.21.15.163:0/645977058 >> 172.21.15.163:0/645977058 conn(0x7f53640b0e30 msgr2=0x7f53640b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:37.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 -- 172.21.15.163:0/645977058 shutdown_connections 2024-07-26T06:17:37.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.358+0000 7f5369800640 1 -- 172.21.15.163:0/645977058 wait complete. 2024-07-26T06:17:37.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.360+0000 7f5369800640 1 Processor -- start 2024-07-26T06:17:37.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.360+0000 7f5369800640 1 -- start start 2024-07-26T06:17:37.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5369800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 0x7f5364156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5369800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53640cd000 0x7f536414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5369800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5364152ef0 0x7f53640ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5369800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f536415d6e0 con 0x7f53640b4420 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5369800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f536415d560 con 0x7f5364152ef0 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5369800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f536415d860 con 0x7f53640cd000 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5363e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 0x7f5364156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5363e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 0x7f5364156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48736/0 (socket says 172.21.15.163:48736) 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.361+0000 7f5363e00640 1 -- 172.21.15.163:0/3284531191 learned_addr learned my addr 172.21.15.163:0/3284531191 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f535be00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53640cd000 0x7f536414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5363e00640 1 -- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53640cd000 msgr2=0x7f536414e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5363e00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53640cd000 0x7f536414e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5368e00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5364152ef0 0x7f53640ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5363e00640 1 -- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5364152ef0 msgr2=0x7f53640ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5363e00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5364152ef0 0x7f53640ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5363e00640 1 -- 172.21.15.163:0/3284531191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53641537d0 con 0x7f53640b4420 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5363e00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 0x7f5364156870 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5354009780 tx=0x7f5354032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354040070 con 0x7f53640b4420 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5354005310 con 0x7f53640b4420 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5364153ab0 con 0x7f53640b4420 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354003800 con 0x7f53640b4420 2024-07-26T06:17:37.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.362+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536414e8b0 con 0x7f53640b4420 2024-07-26T06:17:37.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.364+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f536415ce50 con 0x7f53640b4420 2024-07-26T06:17:37.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.365+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f5354039040 con 0x7f53640b4420 2024-07-26T06:17:37.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.366+0000 7f5360c00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5340076630 0x7f5340078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.366+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f53540b8db0 con 0x7f53640b4420 2024-07-26T06:17:37.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.367+0000 7f535be00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5340076630 0x7f5340078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.368+0000 7f535be00640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5340076630 0x7f5340078af0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5364158d50 tx=0x7f534c005c60 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:37.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.370+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f53540bd070 con 0x7f53640b4420 2024-07-26T06:17:37.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.652+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f536414ea80 con 0x7f53640b4420 2024-07-26T06:17:37.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.693+0000 7f5360c00640 1 -- 172.21.15.163:0/3284531191 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v213) ==== 161+0+0 (secure 0 0 0) 0x7f5354082540 con 0x7f53640b4420 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5340076630 msgr2=0x7f5340078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5340076630 0x7f5340078af0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5364158d50 tx=0x7f534c005c60 comp rx=0 tx=0).stop 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 msgr2=0x7f5364156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 0x7f5364156870 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5354009780 tx=0x7f5354032d50 comp rx=0 tx=0).stop 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 shutdown_connections 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f5340076630 0x7f5340078af0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5364152ef0 0x7f53640ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53640cd000 0x7f536414e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 --2- 172.21.15.163:0/3284531191 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53640b4420 0x7f5364156870 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 >> 172.21.15.163:0/3284531191 conn(0x7f53640b0e30 msgr2=0x7f53640b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:37.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 shutdown_connections 2024-07-26T06:17:37.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.696+0000 7f5369800640 1 -- 172.21.15.163:0/3284531191 wait complete. 2024-07-26T06:17:37.716 INFO:tasks.cephfs_test_runner:Validate handcrafted .meta file on legacy subvol root doesn't break the system ... ok 2024-07-26T06:17:37.716 INFO:tasks.cephfs_test_runner:Starting test: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:17:37.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.913+0000 7ff463200640 1 -- 172.21.15.163:0/3939177901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cbad0 msgr2=0x7ff45c0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.913+0000 7ff463200640 1 --2- 172.21.15.163:0/3939177901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cbad0 0x7ff45c0cd530 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0b1ac0 tx=0x7ff45402aa30 comp rx=0 tx=0).stop 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.913+0000 7ff463200640 1 -- 172.21.15.163:0/3939177901 shutdown_connections 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.913+0000 7ff463200640 1 --2- 172.21.15.163:0/3939177901 >> v2:172.21.15.164:3300/0 conn(0x7ff45c0cd000 0x7ff45c0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.913+0000 7ff463200640 1 --2- 172.21.15.163:0/3939177901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cbad0 0x7ff45c0cd530 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.913+0000 7ff463200640 1 -- 172.21.15.163:0/3939177901 >> 172.21.15.163:0/3939177901 conn(0x7ff45c0b0e30 msgr2=0x7ff45c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.914+0000 7ff463200640 1 -- 172.21.15.163:0/3939177901 shutdown_connections 2024-07-26T06:17:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.914+0000 7ff463200640 1 -- 172.21.15.163:0/3939177901 wait complete. 2024-07-26T06:17:37.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.915+0000 7ff463200640 1 Processor -- start 2024-07-26T06:17:37.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.915+0000 7ff463200640 1 -- start start 2024-07-26T06:17:37.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff463200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff45c0b4420 0x7ff45c1f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff463200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 0x7ff45c158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff463200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff45c14bd60 0x7ff45c13f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff463200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c15d6e0 con 0x7ff45c0cd000 2024-07-26T06:17:37.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff463200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c15d560 con 0x7ff45c0b4420 2024-07-26T06:17:37.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff463200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c15d860 con 0x7ff45c14bd60 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.916+0000 7ff461400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 0x7ff45c158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 0x7ff45c158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48750/0 (socket says 172.21.15.163:48750) 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 -- 172.21.15.163:0/76534709 learned_addr learned my addr 172.21.15.163:0/76534709 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff462800640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff45c14bd60 0x7ff45c13f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 -- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff45c14bd60 msgr2=0x7ff45c13f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461e00640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff45c0b4420 0x7ff45c1f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff45c14bd60 0x7ff45c13f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 -- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff45c0b4420 msgr2=0x7ff45c1f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff45c0b4420 0x7ff45c1f4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 -- 172.21.15.163:0/76534709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c15ad30 con 0x7ff45c0cd000 2024-07-26T06:17:37.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.917+0000 7ff461400640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 0x7ff45c158030 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c410 tx=0x7ff44c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:37.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.918+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c017070 con 0x7ff45c0cd000 2024-07-26T06:17:37.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.918+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff44c0035f0 con 0x7ff45c0cd000 2024-07-26T06:17:37.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.918+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c003a80 con 0x7ff45c0cd000 2024-07-26T06:17:37.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.918+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45c15b010 con 0x7ff45c0cd000 2024-07-26T06:17:37.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.918+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45c15b5d0 con 0x7ff45c0cd000 2024-07-26T06:17:37.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.919+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff45c1555e0 con 0x7ff45c0cd000 2024-07-26T06:17:37.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.920+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7ff44c003c20 con 0x7ff45c0cd000 2024-07-26T06:17:37.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.921+0000 7ff452000640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff440076630 0x7ff440078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:37.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.922+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7ff44c096d10 con 0x7ff45c0cd000 2024-07-26T06:17:37.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.922+0000 7ff461e00640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff440076630 0x7ff440078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:37.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.923+0000 7ff461e00640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff440076630 0x7ff440078af0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff454035000 tx=0x7ff454004ab0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:37.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:37.926+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff45c1555e0 con 0x7ff45c0cd000 2024-07-26T06:17:38.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.136+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff45c0af940 con 0x7ff45c0cd000 2024-07-26T06:17:38.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.138+0000 7ff452000640 1 -- 172.21.15.163:0/76534709 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+13046 (secure 0 0 0) 0x7ff44c09c020 con 0x7ff45c0cd000 2024-07-26T06:17:38.183 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:38.183 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":61,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:36.677780+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6835/3071317413":"2024-07-27T06:17:33.658841+0000","172.21.15.164:6834/3071317413":"2024-07-27T06:17:33.658841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:38.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff440076630 msgr2=0x7ff440078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff440076630 0x7ff440078af0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff454035000 tx=0x7ff454004ab0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 msgr2=0x7ff45c158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 0x7ff45c158030 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c410 tx=0x7ff44c00c8d0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 shutdown_connections 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff440076630 0x7ff440078af0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff45c14bd60 0x7ff45c13f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff45c0cd000 0x7ff45c158030 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 --2- 172.21.15.163:0/76534709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff45c0b4420 0x7ff45c1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 >> 172.21.15.163:0/76534709 conn(0x7ff45c0b0e30 msgr2=0x7ff45c15a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.141+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 shutdown_connections 2024-07-26T06:17:38.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.142+0000 7ff463200640 1 -- 172.21.15.163:0/76534709 wait complete. 2024-07-26T06:17:38.186 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:13:21.677101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:17:38.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.395+0000 7fd62ea00640 1 -- 172.21.15.163:0/1519855042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6280ce830 msgr2=0x7fd6280b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.395+0000 7fd62ea00640 1 --2- 172.21.15.163:0/1519855042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6280ce830 0x7fd6280b64c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd6280690b0 tx=0x7fd61802a4d0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.396+0000 7fd62ea00640 1 -- 172.21.15.163:0/1519855042 shutdown_connections 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.396+0000 7fd62ea00640 1 --2- 172.21.15.163:0/1519855042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd6280ce830 0x7fd6280b64c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.396+0000 7fd62ea00640 1 --2- 172.21.15.163:0/1519855042 >> v2:172.21.15.164:3300/0 conn(0x7fd62815d5a0 0x7fd6280ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.396+0000 7fd62ea00640 1 -- 172.21.15.163:0/1519855042 >> 172.21.15.163:0/1519855042 conn(0x7fd6280cfda0 msgr2=0x7fd6280d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.396+0000 7fd62ea00640 1 -- 172.21.15.163:0/1519855042 shutdown_connections 2024-07-26T06:17:38.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.396+0000 7fd62ea00640 1 -- 172.21.15.163:0/1519855042 wait complete. 2024-07-26T06:17:38.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.397+0000 7fd62ea00640 1 Processor -- start 2024-07-26T06:17:38.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.398+0000 7fd62ea00640 1 -- start start 2024-07-26T06:17:38.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.398+0000 7fd62ea00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6280b1e80 0x7fd6280cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.398+0000 7fd62ea00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6280ce830 0x7fd6281f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 0x7fd6280c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62ea00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd6280b5130 con 0x7fd62815d5a0 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62ea00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd6280b4fb0 con 0x7fd6280ce830 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62ea00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd6280b52b0 con 0x7fd6280b1e80 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 0x7fd6280c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 0x7fd6280c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48766/0 (socket says 172.21.15.163:48766) 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 -- 172.21.15.163:0/357607203 learned_addr learned my addr 172.21.15.163:0/357607203 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62d600640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6280b1e80 0x7fd6280cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62cc00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6280ce830 0x7fd6281f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 -- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6280b1e80 msgr2=0x7fd6280cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6280b1e80 0x7fd6280cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 -- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6280ce830 msgr2=0x7fd6281f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6280ce830 0x7fd6281f3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.399+0000 7fd62e000640 1 -- 172.21.15.163:0/357607203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6280ca640 con 0x7fd62815d5a0 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.400+0000 7fd62e000640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 0x7fd6280c9340 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd62000c7b0 tx=0x7fd62000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.400+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620004b60 con 0x7fd62815d5a0 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.400+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd620004df0 con 0x7fd62815d5a0 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.400+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62000aa90 con 0x7fd62815d5a0 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.400+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6280ca920 con 0x7fd62815d5a0 2024-07-26T06:17:38.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.400+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6280c9880 con 0x7fd62815d5a0 2024-07-26T06:17:38.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.401+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6280b5820 con 0x7fd62815d5a0 2024-07-26T06:17:38.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.402+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fd62000ac30 con 0x7fd62815d5a0 2024-07-26T06:17:38.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.403+0000 7fd625600640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd60c076660 0x7fd60c078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.404+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7fd620097e70 con 0x7fd62815d5a0 2024-07-26T06:17:38.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.404+0000 7fd62d600640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd60c076660 0x7fd60c078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.405+0000 7fd62d600640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd60c076660 0x7fd60c078b20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd61c005fd0 tx=0x7fd61c005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:38.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.407+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd620020070 con 0x7fd62815d5a0 2024-07-26T06:17:38.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.596+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd628074680 con 0x7fd62815d5a0 2024-07-26T06:17:38.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.598+0000 7fd625600640 1 -- 172.21.15.163:0/357607203 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7fd620061500 con 0x7fd62815d5a0 2024-07-26T06:17:38.598 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd60c076660 msgr2=0x7fd60c078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd60c076660 0x7fd60c078b20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd61c005fd0 tx=0x7fd61c005850 comp rx=0 tx=0).stop 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 msgr2=0x7fd6280c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 0x7fd6280c9340 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd62000c7b0 tx=0x7fd62000cc70 comp rx=0 tx=0).stop 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 shutdown_connections 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd60c076660 0x7fd60c078b20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd62815d5a0 0x7fd6280c9340 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6280ce830 0x7fd6281f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 --2- 172.21.15.163:0/357607203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6280b1e80 0x7fd6280cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 >> 172.21.15.163:0/357607203 conn(0x7fd6280cfda0 msgr2=0x7fd6280b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:38.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 shutdown_connections 2024-07-26T06:17:38.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.601+0000 7fd62ea00640 1 -- 172.21.15.163:0/357607203 wait complete. 2024-07-26T06:17:38.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-07-26T06:17:38.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.827+0000 7f1993800640 1 -- 172.21.15.163:0/4173452604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 msgr2=0x7f196c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.827+0000 7f1993800640 1 --2- 172.21.15.163:0/4173452604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f196c00b9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f198c0b4080 tx=0x7f196c005830 comp rx=0 tx=0).stop 2024-07-26T06:17:38.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.828+0000 7f1993800640 1 -- 172.21.15.163:0/4173452604 shutdown_connections 2024-07-26T06:17:38.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.828+0000 7f1993800640 1 --2- 172.21.15.163:0/4173452604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f196c00b9c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.828+0000 7f1993800640 1 -- 172.21.15.163:0/4173452604 >> 172.21.15.163:0/4173452604 conn(0x7f198c0b30d0 msgr2=0x7f198c0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:38.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.828+0000 7f1993800640 1 -- 172.21.15.163:0/4173452604 shutdown_connections 2024-07-26T06:17:38.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.828+0000 7f1993800640 1 -- 172.21.15.163:0/4173452604 wait complete. 2024-07-26T06:17:38.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.829+0000 7f1993800640 1 Processor -- start 2024-07-26T06:17:38.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.830+0000 7f1993800640 1 -- start start 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.830+0000 7f1993800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f198c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.830+0000 7f1993800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f198c13be60 0x7f198c15d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1993800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f198c0aa0a0 0x7f198c0aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1993800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f198c141160 con 0x7f196c005210 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1993800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f198c073a70 con 0x7f198c13be60 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1993800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f198c1412e0 con 0x7f198c0aa0a0 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f198c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f198c15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48796/0 (socket says 172.21.15.163:48796) 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 -- 172.21.15.163:0/380191771 learned_addr learned my addr 172.21.15.163:0/380191771 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f198be00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f198c13be60 0x7f198c15d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 -- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f198c0aa0a0 msgr2=0x7f198c0aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1991800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f198c0aa0a0 0x7f198c0aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f198c0aa0a0 0x7f198c0aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 -- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f198c13be60 msgr2=0x7f198c15d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f198c13be60 0x7f198c15d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f1990e00640 1 -- 172.21.15.163:0/380191771 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c15d870 con 0x7f196c005210 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.831+0000 7f198be00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f198c13be60 0x7f198c15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.832+0000 7f1990e00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f198c15cd00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f197c00f7d0 tx=0x7f197c00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.832+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c00bcb0 con 0x7f196c005210 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.832+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f197c004030 con 0x7f196c005210 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.832+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198c15db50 con 0x7f196c005210 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.832+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c004480 con 0x7f196c005210 2024-07-26T06:17:38.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.832+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c15e110 con 0x7f196c005210 2024-07-26T06:17:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.833+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198c070cb0 con 0x7f196c005210 2024-07-26T06:17:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.834+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f197c0035f0 con 0x7f196c005210 2024-07-26T06:17:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.835+0000 7f1988c00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1968076630 0x7f1968078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.836+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f197c098360 con 0x7f196c005210 2024-07-26T06:17:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.836+0000 7f198be00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1968076630 0x7f1968078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.837+0000 7f198be00640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1968076630 0x7f1968078af0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1974002230 tx=0x7f19740019d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:38.840+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f197c061450 con 0x7f196c005210 2024-07-26T06:17:39.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.129+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f198c13e510 con 0x7f196c005210 2024-07-26T06:17:39.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.706+0000 7f1988c00640 1 -- 172.21.15.163:0/380191771 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v215) ==== 134+0+0 (secure 0 0 0) 0x7f197c065750 con 0x7f196c005210 2024-07-26T06:17:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1968076630 msgr2=0x7f1968078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1968076630 0x7f1968078af0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1974002230 tx=0x7f19740019d0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 msgr2=0x7f198c15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:39.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f198c15cd00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f197c00f7d0 tx=0x7f197c00fc90 comp rx=0 tx=0).stop 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 shutdown_connections 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1968076630 0x7f1968078af0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f198c0aa0a0 0x7f198c0aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f198c13be60 0x7f198c15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 --2- 172.21.15.163:0/380191771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f196c005210 0x7f198c15cd00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.710+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 >> 172.21.15.163:0/380191771 conn(0x7f198c0b30d0 msgr2=0x7f198c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.711+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 shutdown_connections 2024-07-26T06:17:39.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.711+0000 7f1993800640 1 -- 172.21.15.163:0/380191771 wait complete. 2024-07-26T06:17:39.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-07-26T06:17:39.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 -- 172.21.15.163:0/2076578391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4280ce2d0 msgr2=0x7fe428071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:39.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 --2- 172.21.15.163:0/2076578391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4280ce2d0 0x7fe428071530 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe42806ce80 tx=0x7fe41402a560 comp rx=0 tx=0).stop 2024-07-26T06:17:39.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 -- 172.21.15.163:0/2076578391 shutdown_connections 2024-07-26T06:17:39.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 --2- 172.21.15.163:0/2076578391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe428071a70 0x7fe42815ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 --2- 172.21.15.163:0/2076578391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4280ce2d0 0x7fe428071530 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 --2- 172.21.15.163:0/2076578391 >> v2:172.21.15.164:3300/0 conn(0x7fe4281556f0 0x7fe4280cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.938+0000 7fe42e200640 1 -- 172.21.15.163:0/2076578391 >> 172.21.15.163:0/2076578391 conn(0x7fe4280b0e30 msgr2=0x7fe4280b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:39.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.939+0000 7fe42e200640 1 -- 172.21.15.163:0/2076578391 shutdown_connections 2024-07-26T06:17:39.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.939+0000 7fe42e200640 1 -- 172.21.15.163:0/2076578391 wait complete. 2024-07-26T06:17:39.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.940+0000 7fe42e200640 1 Processor -- start 2024-07-26T06:17:39.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.940+0000 7fe42e200640 1 -- start start 2024-07-26T06:17:39.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.941+0000 7fe42e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe428071a70 0x7fe42813e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:39.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.941+0000 7fe42e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4280ce2d0 0x7fe42813dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:39.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.941+0000 7fe42e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 0x7fe4281549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:39.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe42815f2c0 con 0x7fe4281556f0 2024-07-26T06:17:39.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe42815f140 con 0x7fe428071a70 2024-07-26T06:17:39.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe42815f440 con 0x7fe4280ce2d0 2024-07-26T06:17:39.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 0x7fe4281549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:39.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 0x7fe4281549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48822/0 (socket says 172.21.15.163:48822) 2024-07-26T06:17:39.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 -- 172.21.15.163:0/395429137 learned_addr learned my addr 172.21.15.163:0/395429137 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:39.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42ce00640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe428071a70 0x7fe42813e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:39.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe427e00640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4280ce2d0 0x7fe42813dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 -- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4280ce2d0 msgr2=0x7fe42813dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4280ce2d0 0x7fe42813dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 -- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe428071a70 msgr2=0x7fe42813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe428071a70 0x7fe42813e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:39.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.942+0000 7fe42d800640 1 -- 172.21.15.163:0/395429137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe428154f20 con 0x7fe4281556f0 2024-07-26T06:17:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.943+0000 7fe42d800640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 0x7fe4281549e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe42000ca40 tx=0x7fe42000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.943+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42000ad60 con 0x7fe4281556f0 2024-07-26T06:17:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.943+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe420004b20 con 0x7fe4281556f0 2024-07-26T06:17:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.943+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe428152090 con 0x7fe4281556f0 2024-07-26T06:17:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.943+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420005020 con 0x7fe4281556f0 2024-07-26T06:17:39.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.943+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42814c6e0 con 0x7fe4281556f0 2024-07-26T06:17:39.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.944+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42815f940 con 0x7fe4281556f0 2024-07-26T06:17:39.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.946+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fe4200051c0 con 0x7fe4281556f0 2024-07-26T06:17:39.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.947+0000 7fe424c00640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe408076630 0x7fe408078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:39.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.947+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7fe420098220 con 0x7fe4281556f0 2024-07-26T06:17:39.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.947+0000 7fe42ce00640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe408076630 0x7fe408078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:39.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.948+0000 7fe42ce00640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe408076630 0x7fe408078af0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe41c005d70 tx=0x7fe41c005d00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:39.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:39.951+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe428154f20 con 0x7fe4281556f0 2024-07-26T06:17:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.154+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fe4280b34f0 con 0x7fe4281556f0 2024-07-26T06:17:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.155+0000 7fe424c00640 1 -- 172.21.15.163:0/395429137 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v33) ==== 59+0+1998 (secure 0 0 0) 0x7fe420014030 con 0x7fe4281556f0 2024-07-26T06:17:40.155 INFO:teuthology.orchestra.run.smithi163.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-07-26T06:17:40.155 INFO:teuthology.orchestra.run.smithi163.stdout:mon advanced mon_op_complaint_time 120 2024-07-26T06:17:40.155 INFO:teuthology.orchestra.run.smithi163.stdout:mon dev mon_warn_on_pool_no_app false 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd advanced osd_op_complaint_time 180.000000 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 5439.226762 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3901.149445 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5414.054049 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4219.540367 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4287.026771 2024-07-26T06:17:40.156 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4403.390021 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3932.844298 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4626.659828 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds 20/20 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds_balancer 20/20 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_ms 1/1 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_frag true 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_scatterstat true 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced mds_op_complaint_time 180.000000 2024-07-26T06:17:40.157 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_verify_scatter true 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced osd_op_complaint_time 180.000000 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_mon_op_timeout 900 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_osd_op_timeout 900 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced client_mount_timeout 600 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_client 20/20 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_ms 1/1 2024-07-26T06:17:40.158 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_mon_op_timeout 900 2024-07-26T06:17:40.159 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_osd_op_timeout 900 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe408076630 msgr2=0x7fe408078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe408076630 0x7fe408078af0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe41c005d70 tx=0x7fe41c005d00 comp rx=0 tx=0).stop 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 msgr2=0x7fe4281549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 0x7fe4281549e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe42000ca40 tx=0x7fe42000cf00 comp rx=0 tx=0).stop 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 shutdown_connections 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe408076630 0x7fe408078af0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4281556f0 0x7fe4281549e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4280ce2d0 0x7fe42813dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 --2- 172.21.15.163:0/395429137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe428071a70 0x7fe42813e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.158+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 >> 172.21.15.163:0/395429137 conn(0x7fe4280b0e30 msgr2=0x7fe42815c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.159+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 shutdown_connections 2024-07-26T06:17:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.159+0000 7fe42e200640 1 -- 172.21.15.163:0/395429137 wait complete. 2024-07-26T06:17:40.178 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:17:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.383+0000 7f88cda00640 1 -- 172.21.15.163:0/1265735554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c80ce2d0 msgr2=0x7f88c8071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.383+0000 7f88cda00640 1 --2- 172.21.15.163:0/1265735554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c80ce2d0 0x7f88c8071530 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009830 tx=0x7f88b802a510 comp rx=0 tx=0).stop 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.383+0000 7f88cda00640 1 -- 172.21.15.163:0/1265735554 shutdown_connections 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.383+0000 7f88cda00640 1 --2- 172.21.15.163:0/1265735554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c80ce2d0 0x7f88c8071530 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.383+0000 7f88cda00640 1 --2- 172.21.15.163:0/1265735554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c81556f0 0x7f88c80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.383+0000 7f88cda00640 1 -- 172.21.15.163:0/1265735554 >> 172.21.15.163:0/1265735554 conn(0x7f88c80b0e30 msgr2=0x7f88c80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.384+0000 7f88cda00640 1 -- 172.21.15.163:0/1265735554 shutdown_connections 2024-07-26T06:17:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.384+0000 7f88cda00640 1 -- 172.21.15.163:0/1265735554 wait complete. 2024-07-26T06:17:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.386+0000 7f88cda00640 1 Processor -- start 2024-07-26T06:17:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.386+0000 7f88cda00640 1 -- start start 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88cda00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 0x7f88c81f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88cda00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c80ce2d0 0x7f88c814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88cda00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81556f0 0x7f88c814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88cda00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f88c815f310 con 0x7f88c8071a70 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88cda00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f88c8071ee0 con 0x7f88c81556f0 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88cda00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88c815f490 con 0x7f88c80ce2d0 2024-07-26T06:17:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88c7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 0x7f88c81f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:40.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88c7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 0x7f88c81f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48848/0 (socket says 172.21.15.163:48848) 2024-07-26T06:17:40.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.387+0000 7f88c7e00640 1 -- 172.21.15.163:0/3398754154 learned_addr learned my addr 172.21.15.163:0/3398754154 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88bfe00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c80ce2d0 0x7f88c814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88cd000640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81556f0 0x7f88c814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c7e00640 1 -- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c80ce2d0 msgr2=0x7f88c814e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c7e00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c80ce2d0 0x7f88c814e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c7e00640 1 -- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81556f0 msgr2=0x7f88c814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c7e00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81556f0 0x7f88c814e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c7e00640 1 -- 172.21.15.163:0/3398754154 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c8150880 con 0x7f88c8071a70 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c7e00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 0x7f88c81f71d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f88b000d740 tx=0x7f88b000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0017070 con 0x7f88c8071a70 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88b0009d20 con 0x7f88c8071a70 2024-07-26T06:17:40.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.388+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b00106c0 con 0x7f88c8071a70 2024-07-26T06:17:40.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.389+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c814ec30 con 0x7f88c8071a70 2024-07-26T06:17:40.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.389+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c814f1f0 con 0x7f88c8071a70 2024-07-26T06:17:40.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.390+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c8071530 con 0x7f88c8071a70 2024-07-26T06:17:40.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.391+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f88b001a070 con 0x7f88c8071a70 2024-07-26T06:17:40.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.392+0000 7f88c4c00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88a4076630 0x7f88a4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:40.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.392+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f88b0097d00 con 0x7f88c8071a70 2024-07-26T06:17:40.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.392+0000 7f88bfe00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88a4076630 0x7f88a4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:40.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.394+0000 7f88bfe00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88a4076630 0x7f88a4078af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88b8034000 tx=0x7f88b8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:40.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.396+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88b0061300 con 0x7f88c8071a70 2024-07-26T06:17:40.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.607+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f88c80b34f0 con 0x7f88c8071a70 2024-07-26T06:17:40.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.614+0000 7f88c4c00640 1 -- 172.21.15.163:0/3398754154 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f88b0060c00 con 0x7f88c8071a70 2024-07-26T06:17:40.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88a4076630 msgr2=0x7f88a4078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88a4076630 0x7f88a4078af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88b8034000 tx=0x7f88b8004730 comp rx=0 tx=0).stop 2024-07-26T06:17:40.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 msgr2=0x7f88c81f71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:40.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 0x7f88c81f71d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f88b000d740 tx=0x7f88b000dc00 comp rx=0 tx=0).stop 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 shutdown_connections 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f88a4076630 0x7f88a4078af0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81556f0 0x7f88c814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c80ce2d0 0x7f88c814e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 --2- 172.21.15.163:0/3398754154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88c8071a70 0x7f88c81f71d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 >> 172.21.15.163:0/3398754154 conn(0x7f88c80b0e30 msgr2=0x7f88c80b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 shutdown_connections 2024-07-26T06:17:40.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:40.618+0000 7f88cda00640 1 -- 172.21.15.163:0/3398754154 wait complete. 2024-07-26T06:17:40.638 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:17:40.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:17:40.748 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:17:40.877 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:17:40.877 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:40.877 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:40.937 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:17:40.938 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:40.938 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:40.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:17:40.993 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:40.993 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:41.050 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:17:41.051 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:41.051 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:41.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:17:41.110 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-07-26T06:17:41.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:41.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 -- 172.21.15.163:0/3229545553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 msgr2=0x7ff3300cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 --2- 172.21.15.163:0/3229545553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff3300cbab0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff33006f590 tx=0x7ff31c02a560 comp rx=0 tx=0).stop 2024-07-26T06:17:41.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 -- 172.21.15.163:0/3229545553 shutdown_connections 2024-07-26T06:17:41.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 --2- 172.21.15.163:0/3229545553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 0x7ff33015d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 --2- 172.21.15.163:0/3229545553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff3300cbab0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 --2- 172.21.15.163:0/3229545553 >> v2:172.21.15.164:3300/0 conn(0x7ff3300741f0 0x7ff3300cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 -- 172.21.15.163:0/3229545553 >> 172.21.15.163:0/3229545553 conn(0x7ff3300b0e30 msgr2=0x7ff3300b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:41.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 -- 172.21.15.163:0/3229545553 shutdown_connections 2024-07-26T06:17:41.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.327+0000 7ff337000640 1 -- 172.21.15.163:0/3229545553 wait complete. 2024-07-26T06:17:41.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.329+0000 7ff337000640 1 Processor -- start 2024-07-26T06:17:41.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.329+0000 7ff337000640 1 -- start start 2024-07-26T06:17:41.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff337000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3300741f0 0x7ff3301e5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff337000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 0x7ff3300adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff337000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff33014a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff337000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff33015d8e0 con 0x7ff33015a6b0 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff337000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff33015d760 con 0x7ff3300741f0 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff337000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff33015da60 con 0x7ff3300cbff0 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.330+0000 7ff336600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff33014a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff33014a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48864/0 (socket says 172.21.15.163:48864) 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 -- 172.21.15.163:0/61024202 learned_addr learned my addr 172.21.15.163:0/61024202 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff335200640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 0x7ff3300adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff335c00640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3300741f0 0x7ff3301e5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 -- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 msgr2=0x7ff3300adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 0x7ff3300adb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 -- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3300741f0 msgr2=0x7ff3301e5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3300741f0 0x7ff3301e5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 -- 172.21.15.163:0/61024202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3301f76e0 con 0x7ff33015a6b0 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff335200640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 0x7ff3300adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff335c00640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3300741f0 0x7ff3301e5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff336600640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff33014a680 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b9a0 tx=0x7ff32800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.331+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328019070 con 0x7ff33015a6b0 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.332+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff32800ccb0 con 0x7ff33015a6b0 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.332+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3280146a0 con 0x7ff33015a6b0 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.332+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3301f79c0 con 0x7ff33015a6b0 2024-07-26T06:17:41.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.332+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301f7f80 con 0x7ff33015a6b0 2024-07-26T06:17:41.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.333+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33015d220 con 0x7ff33015a6b0 2024-07-26T06:17:41.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.334+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7ff32801c070 con 0x7ff33015a6b0 2024-07-26T06:17:41.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.335+0000 7ff32d600640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff310076630 0x7ff310078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.335+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7ff328097730 con 0x7ff33015a6b0 2024-07-26T06:17:41.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.336+0000 7ff335c00640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff310076630 0x7ff310078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.337+0000 7ff335c00640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff310076630 0x7ff310078af0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff32400a870 tx=0x7ff324008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:41.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.339+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff328061010 con 0x7ff33015a6b0 2024-07-26T06:17:41.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.586+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3300b4aa0 con 0x7ff33015a6b0 2024-07-26T06:17:41.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.587+0000 7ff32d600640 1 -- 172.21.15.163:0/61024202 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+3920 (secure 0 0 0) 0x7ff328060910 con 0x7ff33015a6b0 2024-07-26T06:17:41.587 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:41.587 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":38,"btime":"2024-07-26T06:17:34:663461+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4832,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3804200355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3804200355},{"type":"v1","addr":"172.21.15.163:6834","nonce":3804200355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":4835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/3341177733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":3341177733},{"type":"v1","addr":"172.21.15.163:6835","nonce":3341177733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/302849354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":302849354},{"type":"v1","addr":"172.21.15.164:6837","nonce":302849354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":4958,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3572993646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3572993646},{"type":"v1","addr":"172.21.15.164:6835","nonce":3572993646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[]} 2024-07-26T06:17:41.588 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 38 2024-07-26T06:17:41.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff310076630 msgr2=0x7ff310078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff310076630 0x7ff310078af0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff32400a870 tx=0x7ff324008040 comp rx=0 tx=0).stop 2024-07-26T06:17:41.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 msgr2=0x7ff33014a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff33014a680 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b9a0 tx=0x7ff32800be60 comp rx=0 tx=0).stop 2024-07-26T06:17:41.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 shutdown_connections 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff310076630 0x7ff310078af0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff33015a6b0 0x7ff33014a680 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3300cbff0 0x7ff3300adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 --2- 172.21.15.163:0/61024202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3300741f0 0x7ff3301e5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 >> 172.21.15.163:0/61024202 conn(0x7ff3300b0e30 msgr2=0x7ff330071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.590+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 shutdown_connections 2024-07-26T06:17:41.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.591+0000 7ff337000640 1 -- 172.21.15.163:0/61024202 wait complete. 2024-07-26T06:17:41.611 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:17:41.611 INFO:tasks.ceph.mds.a:Stopping old one... 2024-07-26T06:17:41.611 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:17:41.611 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:17:41.611 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:17:41.611 INFO:tasks.ceph.mds.c:Stopping old one... 2024-07-26T06:17:41.612 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:17:41.612 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:17:41.612 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:17:41.612 INFO:tasks.ceph.mds.b:Stopping old one... 2024-07-26T06:17:41.612 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:17:41.612 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:17:41.612 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:17:41.612 INFO:tasks.ceph.mds.d:Stopping old one... 2024-07-26T06:17:41.613 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:17:41.613 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:17:41.629 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:17:41.629 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:17:41.633 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:17:41.634 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:17:41.638 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:17:41.638 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:17:41.639 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:17:41.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:17:41.683 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:17:41.683 DEBUG:teuthology.parallel:result is None 2024-07-26T06:17:41.688 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:17:41.688 DEBUG:teuthology.parallel:result is None 2024-07-26T06:17:41.689 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:17:41.689 DEBUG:teuthology.parallel:result is None 2024-07-26T06:17:41.694 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:17:41.694 DEBUG:teuthology.parallel:result is None 2024-07-26T06:17:41.694 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-07-26T06:17:41.832 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:17:41.833 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:17:41.845 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:17:41.846 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:17:41.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 -- 172.21.15.163:0/271764869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b1e80 msgr2=0x7f9f0c0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 --2- 172.21.15.163:0/271764869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b1e80 0x7f9f0c0b64c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8009860 tx=0x7f9ef802a510 comp rx=0 tx=0).stop 2024-07-26T06:17:41.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 -- 172.21.15.163:0/271764869 shutdown_connections 2024-07-26T06:17:41.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 --2- 172.21.15.163:0/271764869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b1e80 0x7f9f0c0b64c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 -- 172.21.15.163:0/271764869 >> 172.21.15.163:0/271764869 conn(0x7f9f0c0cfda0 msgr2=0x7f9f0c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:41.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 -- 172.21.15.163:0/271764869 shutdown_connections 2024-07-26T06:17:41.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.880+0000 7f9f12400640 1 -- 172.21.15.163:0/271764869 wait complete. 2024-07-26T06:17:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.881+0000 7f9f12400640 1 Processor -- start 2024-07-26T06:17:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.882+0000 7f9f12400640 1 -- start start 2024-07-26T06:17:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.882+0000 7f9f12400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 0x7f9f0c0c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f12400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b23d0 0x7f9f0c1519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f12400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f0c15d5a0 0x7f9f0c0b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f12400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c0b5180 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f12400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c0b5000 con 0x7f9f0c15d5a0 2024-07-26T06:17:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f12400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9f0c0b5300 con 0x7f9f0c0b23d0 2024-07-26T06:17:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 0x7f9f0c0c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 0x7f9f0c0c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48900/0 (socket says 172.21.15.163:48900) 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 -- 172.21.15.163:0/4060516532 learned_addr learned my addr 172.21.15.163:0/4060516532 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f0be00640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b23d0 0x7f9f0c1519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11a00640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f0c15d5a0 0x7f9f0c0b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 -- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b23d0 msgr2=0x7f9f0c1519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b23d0 0x7f9f0c1519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 -- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f0c15d5a0 msgr2=0x7f9f0c0b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f0c15d5a0 0x7f9f0c0b7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.883+0000 7f9f11000640 1 -- 172.21.15.163:0/4060516532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c0b8440 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.884+0000 7f9f11000640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 0x7f9f0c0c9510 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000c460 tx=0x7f9f0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.884+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00016020 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.884+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f00012960 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.884+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c0c9b60 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.884+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00012db0 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.884+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c156fb0 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.885+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f0c1574a0 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.886+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f9f00019020 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.888+0000 7f9f08c00640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9eec076630 0x7f9eec078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.888+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f9f000971d0 con 0x7f9f0c0b1e80 2024-07-26T06:17:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.889+0000 7f9f0be00640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9eec076630 0x7f9eec078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.889+0000 7f9f0be00640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9eec076630 0x7f9eec078af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8034000 tx=0x7f9ef8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:41.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:41.892+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f00020070 con 0x7f9f0c0b1e80 2024-07-26T06:17:42.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.112+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9f0c151f60 con 0x7f9f0c0b1e80 2024-07-26T06:17:42.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.742+0000 7f9f08c00640 1 -- 172.21.15.163:0/4060516532 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) ==== 97+0+0 (secure 0 0 0) 0x7f9f00095e20 con 0x7f9f0c0b1e80 2024-07-26T06:17:42.742 INFO:teuthology.orchestra.run.smithi163.stderr: removed all blocklist entries 2024-07-26T06:17:42.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9eec076630 msgr2=0x7f9eec078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9eec076630 0x7f9eec078af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8034000 tx=0x7f9ef8004730 comp rx=0 tx=0).stop 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 msgr2=0x7f9f0c0c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 0x7f9f0c0c9510 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000c460 tx=0x7f9f0000c920 comp rx=0 tx=0).stop 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 shutdown_connections 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9eec076630 0x7f9eec078af0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f0c15d5a0 0x7f9f0c0b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0c0b23d0 0x7f9f0c1519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 --2- 172.21.15.163:0/4060516532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9f0c0b1e80 0x7f9f0c0c9510 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.745+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 >> 172.21.15.163:0/4060516532 conn(0x7f9f0c0cfda0 msgr2=0x7f9f0c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.746+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 shutdown_connections 2024-07-26T06:17:42.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.746+0000 7f9f12400640 1 -- 172.21.15.163:0/4060516532 wait complete. 2024-07-26T06:17:42.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-07-26T06:17:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57b200640 1 -- 172.21.15.163:0/1015765949 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 msgr2=0x7ff56000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57b200640 1 --2- 172.21.15.163:0/1015765949 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff56000b9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff574009e20 tx=0x7ff560005830 comp rx=0 tx=0).stop 2024-07-26T06:17:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57a800640 1 -- 172.21.15.163:0/1015765949 reap_dead start 2024-07-26T06:17:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57b200640 1 -- 172.21.15.163:0/1015765949 shutdown_connections 2024-07-26T06:17:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57b200640 1 -- 172.21.15.163:0/1015765949 >> 172.21.15.163:0/1015765949 conn(0x7ff57c0cfda0 msgr2=0x7ff57c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:42.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57b200640 1 -- 172.21.15.163:0/1015765949 shutdown_connections 2024-07-26T06:17:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.960+0000 7ff57b200640 1 -- 172.21.15.163:0/1015765949 wait complete. 2024-07-26T06:17:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.962+0000 7ff57b200640 1 Processor -- start 2024-07-26T06:17:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.962+0000 7ff57b200640 1 -- start start 2024-07-26T06:17:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff57b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff57c0c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff57b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff57c14b620 0x7ff57c15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff57b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff57c159670 0x7ff57c159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff57b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c0b6f60 con 0x7ff560004f40 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff57b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c0b6de0 con 0x7ff57c14b620 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff57b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff57c0b4da0 con 0x7ff57c159670 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff579e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff57c0c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff579e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff57c0c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48912/0 (socket says 172.21.15.163:48912) 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff579e00640 1 -- 172.21.15.163:0/3415388456 learned_addr learned my addr 172.21.15.163:0/3415388456 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.963+0000 7ff573e00640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff57c14b620 0x7ff57c15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff579e00640 1 -- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff57c159670 msgr2=0x7ff57c159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff57a800640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff57c159670 0x7ff57c159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff579e00640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff57c159670 0x7ff57c159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff579e00640 1 -- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff57c14b620 msgr2=0x7ff57c15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:42.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff579e00640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff57c14b620 0x7ff57c15c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff579e00640 1 -- 172.21.15.163:0/3415388456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c1f3a10 con 0x7ff560004f40 2024-07-26T06:17:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff579e00640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff57c0c6e50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00d7d0 tx=0x7ff56c00dbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c018070 con 0x7ff560004f40 2024-07-26T06:17:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff56c0050d0 con 0x7ff560004f40 2024-07-26T06:17:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00b6c0 con 0x7ff560004f40 2024-07-26T06:17:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c155de0 con 0x7ff560004f40 2024-07-26T06:17:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.964+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57c0c7710 con 0x7ff560004f40 2024-07-26T06:17:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.965+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57c0b64c0 con 0x7ff560004f40 2024-07-26T06:17:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.967+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7ff56c01b070 con 0x7ff560004f40 2024-07-26T06:17:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.968+0000 7ff572000640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff558076630 0x7ff558078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.968+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4857+0+0 (secure 0 0 0) 0x7ff56c097430 con 0x7ff560004f40 2024-07-26T06:17:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.968+0000 7ff573e00640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff558076630 0x7ff558078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:42.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.969+0000 7ff573e00640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff558076630 0x7ff558078af0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff57c150ff0 tx=0x7ff564006b80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:42.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:42.972+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff56c0604d0 con 0x7ff560004f40 2024-07-26T06:17:43.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.254+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff57c155fb0 con 0x7ff560004f40 2024-07-26T06:17:43.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.255+0000 7ff572000640 1 -- 172.21.15.163:0/3415388456 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5617 (secure 0 0 0) 0x7ff56c0647d0 con 0x7ff560004f40 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_dump": [ 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.a", 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA==", 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.256 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.b", 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg==", 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.c", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw==", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.d", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg==", 2024-07-26T06:17:43.258 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.0", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA==", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.1", 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA==", 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.2", 2024-07-26T06:17:43.260 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA==", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.3", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg==", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.4", 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA==", 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.5", 2024-07-26T06:17:43.262 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g==", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.6", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ==", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.263 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.264 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.7", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA==", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.0", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew==", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.265 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.1", 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ==", 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.2", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw==", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.3", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw==", 2024-07-26T06:17:43.267 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.269 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:17:43.269 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.admin", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm+ibKBxAA1ZBGvVy5rNIzMWwOFwFJFw==", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.270 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.alice", 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBwP6NmeYMJAhAA1U3hd/zPtWuZfG50TBHX5w==", 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol84520", 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw pool=cephfs_data" 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.271 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.x", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ==", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.y", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ==", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:17:43.272 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:17:43.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:43.273 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:17:43.273 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff558076630 msgr2=0x7ff558078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff558076630 0x7ff558078af0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff57c150ff0 tx=0x7ff564006b80 comp rx=0 tx=0).stop 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 msgr2=0x7ff57c0c6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff57c0c6e50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00d7d0 tx=0x7ff56c00dbd0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 shutdown_connections 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7ff558076630 0x7ff558078af0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff57c159670 0x7ff57c159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff57c14b620 0x7ff57c15c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 --2- 172.21.15.163:0/3415388456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff560004f40 0x7ff57c0c6e50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 >> 172.21.15.163:0/3415388456 conn(0x7ff57c0cfda0 msgr2=0x7ff57c0b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:43.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 shutdown_connections 2024-07-26T06:17:43.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.258+0000 7ff57b200640 1 -- 172.21.15.163:0/3415388456 wait complete. 2024-07-26T06:17:43.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.alice 2024-07-26T06:17:43.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 -- 172.21.15.163:0/3775403217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 msgr2=0x7fd9780cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 --2- 172.21.15.163:0/3775403217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd9780cdd90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd968005fd0 tx=0x7fd96802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 -- 172.21.15.163:0/3775403217 shutdown_connections 2024-07-26T06:17:43.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 --2- 172.21.15.163:0/3775403217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd9780cdd90 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 -- 172.21.15.163:0/3775403217 >> 172.21.15.163:0/3775403217 conn(0x7fd9780b0e30 msgr2=0x7fd9780b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:43.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 -- 172.21.15.163:0/3775403217 shutdown_connections 2024-07-26T06:17:43.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.495+0000 7fd97e200640 1 -- 172.21.15.163:0/3775403217 wait complete. 2024-07-26T06:17:43.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.497+0000 7fd97e200640 1 Processor -- start 2024-07-26T06:17:43.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.497+0000 7fd97e200640 1 -- start start 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd978071c40 0x7fd9781493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9780ce2d0 0x7fd978145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd97814c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd9780b41f0 con 0x7fd9781556f0 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd9780b4070 con 0x7fd9780ce2d0 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd9780b4370 con 0x7fd978071c40 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd97814c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:43.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd97814c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48930/0 (socket says 172.21.15.163:48930) 2024-07-26T06:17:43.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.498+0000 7fd97d800640 1 -- 172.21.15.163:0/2427056133 learned_addr learned my addr 172.21.15.163:0/2427056133 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd96fe00640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9780ce2d0 0x7fd978145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97ce00640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd978071c40 0x7fd9781493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97d800640 1 -- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd978071c40 msgr2=0x7fd9781493e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97d800640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd978071c40 0x7fd9781493e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97d800640 1 -- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9780ce2d0 msgr2=0x7fd978145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97d800640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9780ce2d0 0x7fd978145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97d800640 1 -- 172.21.15.163:0/2427056133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97814cd90 con 0x7fd9781556f0 2024-07-26T06:17:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd96fe00640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9780ce2d0 0x7fd978145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:43.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97d800640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd97814c1d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd97000c7a0 tx=0x7fd97000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970003b70 con 0x7fd9781556f0 2024-07-26T06:17:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd970003d30 con 0x7fd9781556f0 2024-07-26T06:17:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97000aa50 con 0x7fd9781556f0 2024-07-26T06:17:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.499+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97814cab0 con 0x7fd9781556f0 2024-07-26T06:17:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.500+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9781f43a0 con 0x7fd9781556f0 2024-07-26T06:17:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.501+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9780cdd90 con 0x7fd9781556f0 2024-07-26T06:17:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.502+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fd97000abf0 con 0x7fd9781556f0 2024-07-26T06:17:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.503+0000 7fd975600640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd954076630 0x7fd954078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.503+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4857+0+0 (secure 0 0 0) 0x7fd970098350 con 0x7fd9781556f0 2024-07-26T06:17:43.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.504+0000 7fd97ce00640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd954076630 0x7fd954078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:43.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.505+0000 7fd97ce00640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd954076630 0x7fd954078af0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd968033000 tx=0x7fd968004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:43.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.506+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd970061bc0 con 0x7fd9781556f0 2024-07-26T06:17:43.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.789+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.alice"} v 0) -- 0x7fd9780b34f0 con 0x7fd9781556f0 2024-07-26T06:17:43.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.793+0000 7fd975600640 1 -- 172.21.15.163:0/2427056133 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.alice"}]=0 v14) ==== 82+0+0 (secure 0 0 0) 0x7fd9700614c0 con 0x7fd9781556f0 2024-07-26T06:17:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.796+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd954076630 msgr2=0x7fd954078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.796+0000 7fd97e200640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd954076630 0x7fd954078af0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd968033000 tx=0x7fd968004730 comp rx=0 tx=0).stop 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.796+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 msgr2=0x7fd97814c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.796+0000 7fd97e200640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd97814c1d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd97000c7a0 tx=0x7fd97000cc60 comp rx=0 tx=0).stop 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 shutdown_connections 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fd954076630 0x7fd954078af0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd9781556f0 0x7fd97814c1d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9780ce2d0 0x7fd978145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 --2- 172.21.15.163:0/2427056133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd978071c40 0x7fd9781493e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 >> 172.21.15.163:0/2427056133 conn(0x7fd9780b0e30 msgr2=0x7fd978150a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 shutdown_connections 2024-07-26T06:17:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:43.797+0000 7fd97e200640 1 -- 172.21.15.163:0/2427056133 wait complete. 2024-07-26T06:17:43.817 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:17:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bd800640 1 -- 172.21.15.163:0/3095764051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 msgr2=0x7f179c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bd800640 1 --2- 172.21.15.163:0/3095764051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f179c003e50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f17a800c310 tx=0x7f179c005830 comp rx=0 tx=0).stop 2024-07-26T06:17:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bce00640 1 -- 172.21.15.163:0/3095764051 reap_dead start 2024-07-26T06:17:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bd800640 1 -- 172.21.15.163:0/3095764051 shutdown_connections 2024-07-26T06:17:44.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bd800640 1 -- 172.21.15.163:0/3095764051 >> 172.21.15.163:0/3095764051 conn(0x7f17b80b0e30 msgr2=0x7f17b80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:44.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bd800640 1 -- 172.21.15.163:0/3095764051 shutdown_connections 2024-07-26T06:17:44.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.010+0000 7f17bd800640 1 -- 172.21.15.163:0/3095764051 wait complete. 2024-07-26T06:17:44.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.012+0000 7f17bd800640 1 Processor -- start 2024-07-26T06:17:44.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.012+0000 7f17bd800640 1 -- start start 2024-07-26T06:17:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17bd800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f17b815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17bd800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17b8157480 0x7f17b81f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17bd800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17b80afdc0 0x7f17b80b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17bd800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17b806f470 con 0x7f179c003a30 2024-07-26T06:17:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17bd800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17b806f2f0 con 0x7f17b8157480 2024-07-26T06:17:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17bd800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17b8159750 con 0x7f17b80afdc0 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f17b815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f17b815b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48954/0 (socket says 172.21.15.163:48954) 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 -- 172.21.15.163:0/2471353603 learned_addr learned my addr 172.21.15.163:0/2471353603 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 -- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17b80afdc0 msgr2=0x7f17b80b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17b80afdc0 0x7f17b80b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 -- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17b8157480 msgr2=0x7f17b81f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17b8157480 0x7f17b81f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.013+0000 7f17b7e00640 1 -- 172.21.15.163:0/2471353603 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b8157bf0 con 0x7f179c003a30 2024-07-26T06:17:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.014+0000 7f17b7e00640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f17b815b6f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f17a802b180 tx=0x7f17a8004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.014+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a8031040 con 0x7f179c003a30 2024-07-26T06:17:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.014+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17a8006420 con 0x7f179c003a30 2024-07-26T06:17:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.014+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a8006c90 con 0x7f179c003a30 2024-07-26T06:17:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.014+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b8157ed0 con 0x7f179c003a30 2024-07-26T06:17:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.014+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b8158490 con 0x7f179c003a30 2024-07-26T06:17:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.015+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b80cd530 con 0x7f179c003a30 2024-07-26T06:17:44.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.016+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f17a8034070 con 0x7f179c003a30 2024-07-26T06:17:44.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.018+0000 7f17b4c00640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1794076660 0x7f1794078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.018+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4857+0+0 (secure 0 0 0) 0x7f17a80b7cc0 con 0x7f179c003a30 2024-07-26T06:17:44.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.018+0000 7f17af400640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1794076660 0x7f1794078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:44.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.020+0000 7f17af400640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1794076660 0x7f1794078b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f17b8151bc0 tx=0x7f17a0007970 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:44.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.022+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f17a80be080 con 0x7f179c003a30 2024-07-26T06:17:44.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.216+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f17b8153080 con 0x7f179c003a30 2024-07-26T06:17:44.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.217+0000 7f17b4c00640 1 -- 172.21.15.163:0/2471353603 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) ==== 84+0+65 (secure 0 0 0) 0x7f17a8081310 con 0x7f179c003a30 2024-07-26T06:17:44.217 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:44.217 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:17:44.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:44.217 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:17:44.218 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:17:44.218 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:44.218 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:17:44.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1794076660 msgr2=0x7f1794078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1794076660 0x7f1794078b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f17b8151bc0 tx=0x7f17a0007970 comp rx=0 tx=0).stop 2024-07-26T06:17:44.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 msgr2=0x7f17b815b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f17b815b6f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f17a802b180 tx=0x7f17a8004bc0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 shutdown_connections 2024-07-26T06:17:44.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f1794076660 0x7f1794078b20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17b80afdc0 0x7f17b80b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17b8157480 0x7f17b81f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 --2- 172.21.15.163:0/2471353603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f179c003a30 0x7f17b815b6f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 >> 172.21.15.163:0/2471353603 conn(0x7f17b80b0e30 msgr2=0x7f17b80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:44.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 shutdown_connections 2024-07-26T06:17:44.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.220+0000 7f17bd800640 1 -- 172.21.15.163:0/2471353603 wait complete. 2024-07-26T06:17:44.239 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:17:44.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1342755727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00cc450 msgr2=0x7f8bf01681a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1342755727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00cc450 0x7f8bf01681a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8be8005fd0 tx=0x7f8be802a8b0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1342755727 shutdown_connections 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1342755727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00cc450 0x7f8bf01681a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1342755727 >> 172.21.15.163:0/1342755727 conn(0x7f8bf00b0e30 msgr2=0x7f8bf00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1342755727 shutdown_connections 2024-07-26T06:17:44.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.447+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1342755727 wait complete. 2024-07-26T06:17:44.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.448+0000 7f8bf5a00640 1 Processor -- start 2024-07-26T06:17:44.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.449+0000 7f8bf5a00640 1 -- start start 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.449+0000 7f8bf5a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 0x7f8bf0167f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.449+0000 7f8bf5a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8bf00cc450 0x7f8bf013e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8bf5a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8bf0146be0 0x7f8bf00733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8bf5a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf0168860 con 0x7f8bf00adff0 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8bf5a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf01686e0 con 0x7f8bf00cc450 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8bf5a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf01689e0 con 0x7f8bf0146be0 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 0x7f8bf0167f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:44.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 0x7f8bf0167f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:48968/0 (socket says 172.21.15.163:48968) 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 -- 172.21.15.163:0/1858618133 learned_addr learned my addr 172.21.15.163:0/1858618133 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8be7400640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8bf00cc450 0x7f8bf013e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 -- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8bf0146be0 msgr2=0x7f8bf00733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8bf5000640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8bf0146be0 0x7f8bf00733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8bf0146be0 0x7f8bf00733f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 -- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8bf00cc450 msgr2=0x7f8bf013e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8bf00cc450 0x7f8bf013e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8befe00640 1 -- 172.21.15.163:0/1858618133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf013ef00 con 0x7f8bf00adff0 2024-07-26T06:17:44.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.450+0000 7f8be7400640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8bf00cc450 0x7f8bf013e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.451+0000 7f8befe00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 0x7f8bf0167f10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8be000b3f0 tx=0x7f8be000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.451+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0015020 con 0x7f8bf00adff0 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.451+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8be0011920 con 0x7f8bf00adff0 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.451+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0011d70 con 0x7f8bf00adff0 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.451+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf013f1e0 con 0x7f8bf00adff0 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.451+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf013f7a0 con 0x7f8bf00adff0 2024-07-26T06:17:44.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.452+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf013ff80 con 0x7f8bf00adff0 2024-07-26T06:17:44.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.453+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f8be0018020 con 0x7f8bf00adff0 2024-07-26T06:17:44.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.454+0000 7f8becc00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8bcc076630 0x7f8bcc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:44.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.454+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4857+0+0 (secure 0 0 0) 0x7f8be00975b0 con 0x7f8bf00adff0 2024-07-26T06:17:44.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.455+0000 7f8be7400640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8bcc076630 0x7f8bcc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:44.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.456+0000 7f8be7400640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8bcc076630 0x7f8bcc078af0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8007fd0 tx=0x7f8bd8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:44.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.458+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8bf013ff80 con 0x7f8bf00adff0 2024-07-26T06:17:44.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.663+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8bf00ae8b0 con 0x7f8bf00adff0 2024-07-26T06:17:44.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.798+0000 7f8becc00640 1 -- 172.21.15.163:0/1858618133 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) ==== 138+0+0 (secure 0 0 0) 0x7f8be001f030 con 0x7f8bf00adff0 2024-07-26T06:17:44.798 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:17:44.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.802+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8bcc076630 msgr2=0x7f8bcc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.802+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8bcc076630 0x7f8bcc078af0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8007fd0 tx=0x7f8bd8009040 comp rx=0 tx=0).stop 2024-07-26T06:17:44.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.802+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 msgr2=0x7f8bf0167f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:44.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.802+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 0x7f8bf0167f10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8be000b3f0 tx=0x7f8be000b8b0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.803+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 shutdown_connections 2024-07-26T06:17:44.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.803+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f8bcc076630 0x7f8bcc078af0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.803+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8bf0146be0 0x7f8bf00733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.803+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8bf00cc450 0x7f8bf013e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.803+0000 7f8bf5a00640 1 --2- 172.21.15.163:0/1858618133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf00adff0 0x7f8bf0167f10 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:44.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.803+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 >> 172.21.15.163:0/1858618133 conn(0x7f8bf00b0e30 msgr2=0x7f8bf015e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:44.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.804+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 shutdown_connections 2024-07-26T06:17:44.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:44.804+0000 7f8bf5a00640 1 -- 172.21.15.163:0/1858618133 wait complete. 2024-07-26T06:17:44.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:17:45.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.040+0000 7fb120e00640 1 -- 172.21.15.163:0/159203825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0fc0052e0 msgr2=0x7fb0fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.040+0000 7fb120e00640 1 --2- 172.21.15.163:0/159203825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0fc0052e0 0x7fb0fc00b9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb114001690 tx=0x7fb0fc005830 comp rx=0 tx=0).stop 2024-07-26T06:17:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.040+0000 7fb120e00640 1 -- 172.21.15.163:0/159203825 shutdown_connections 2024-07-26T06:17:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.040+0000 7fb120e00640 1 --2- 172.21.15.163:0/159203825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0fc0052e0 0x7fb0fc00b9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.040+0000 7fb120e00640 1 -- 172.21.15.163:0/159203825 >> 172.21.15.163:0/159203825 conn(0x7fb11c0b3060 msgr2=0x7fb11c0b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.040+0000 7fb120e00640 1 -- 172.21.15.163:0/159203825 shutdown_connections 2024-07-26T06:17:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.041+0000 7fb120e00640 1 -- 172.21.15.163:0/159203825 wait complete. 2024-07-26T06:17:45.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.042+0000 7fb120e00640 1 Processor -- start 2024-07-26T06:17:45.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.042+0000 7fb120e00640 1 -- start start 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.043+0000 7fb120e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0fc0052e0 0x7fb11c1562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.043+0000 7fb120e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11c15f130 0x7fb11c15da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.043+0000 7fb120e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 0x7fb11c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.043+0000 7fb120e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c0b4bd0 con 0x7fb11c14dbd0 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.043+0000 7fb120e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c0b4a50 con 0x7fb11c15f130 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.043+0000 7fb120e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb11c0b4d50 con 0x7fb0fc0052e0 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 0x7fb11c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:45.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 0x7fb11c158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34008/0 (socket says 172.21.15.163:34008) 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 -- 172.21.15.163:0/808650119 learned_addr learned my addr 172.21.15.163:0/808650119 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11b400640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0fc0052e0 0x7fb11c1562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11aa00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11c15f130 0x7fb11c15da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 -- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0fc0052e0 msgr2=0x7fb11c1562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0fc0052e0 0x7fb11c1562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 -- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11c15f130 msgr2=0x7fb11c15da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11c15f130 0x7fb11c15da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 -- 172.21.15.163:0/808650119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c160a60 con 0x7fb11c14dbd0 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11b400640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0fc0052e0 0x7fb11c1562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11aa00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11c15f130 0x7fb11c15da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:45.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.044+0000 7fb11be00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 0x7fb11c158340 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb11402a910 tx=0x7fb1140062e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:45.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.045+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1140037f0 con 0x7fb11c14dbd0 2024-07-26T06:17:45.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.045+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb114003990 con 0x7fb11c14dbd0 2024-07-26T06:17:45.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.045+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114003dc0 con 0x7fb11c14dbd0 2024-07-26T06:17:45.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.045+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11c15e090 con 0x7fb11c14dbd0 2024-07-26T06:17:45.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.045+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11c15e650 con 0x7fb11c14dbd0 2024-07-26T06:17:45.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.046+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb11c071060 con 0x7fb11c14dbd0 2024-07-26T06:17:45.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.048+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fb114030020 con 0x7fb11c14dbd0 2024-07-26T06:17:45.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.049+0000 7fb113400640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb0f8076630 0x7fb0f8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:45.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.049+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5227+0+0 (secure 0 0 0) 0x7fb114038020 con 0x7fb11c14dbd0 2024-07-26T06:17:45.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.049+0000 7fb11b400640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb0f8076630 0x7fb0f8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:45.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.051+0000 7fb11b400640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb0f8076630 0x7fb0f8078af0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009b60 tx=0x7fb10c005e70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:45.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.052+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb114080b30 con 0x7fb11c14dbd0 2024-07-26T06:17:45.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.256+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb11c15a900 con 0x7fb11c14dbd0 2024-07-26T06:17:45.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.803+0000 7fb113400640 1 -- 172.21.15.163:0/808650119 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) ==== 170+0+0 (secure 0 0 0) 0x7fb114080430 con 0x7fb11c14dbd0 2024-07-26T06:17:45.805 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:17:45.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.808+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb0f8076630 msgr2=0x7fb0f8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:45.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.808+0000 7fb120e00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb0f8076630 0x7fb0f8078af0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009b60 tx=0x7fb10c005e70 comp rx=0 tx=0).stop 2024-07-26T06:17:45.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.808+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 msgr2=0x7fb11c158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:45.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.808+0000 7fb120e00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 0x7fb11c158340 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb11402a910 tx=0x7fb1140062e0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 shutdown_connections 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb0f8076630 0x7fb0f8078af0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb11c14dbd0 0x7fb11c158340 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11c15f130 0x7fb11c15da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 --2- 172.21.15.163:0/808650119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0fc0052e0 0x7fb11c1562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 >> 172.21.15.163:0/808650119 conn(0x7fb11c0b3060 msgr2=0x7fb11c0af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 shutdown_connections 2024-07-26T06:17:45.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:45.809+0000 7fb120e00640 1 -- 172.21.15.163:0/808650119 wait complete. 2024-07-26T06:17:45.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.026+0000 7f0fdf200640 1 -- 172.21.15.163:0/2900564249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd8159380 msgr2=0x7f0fd80b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.026+0000 7f0fdf200640 1 --2- 172.21.15.163:0/2900564249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd8159380 0x7f0fd80b2f90 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80cb730 tx=0x7f0fc802a560 comp rx=0 tx=0).stop 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.027+0000 7f0fdf200640 1 -- 172.21.15.163:0/2900564249 shutdown_connections 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.027+0000 7f0fdf200640 1 --2- 172.21.15.163:0/2900564249 >> v2:172.21.15.164:3300/0 conn(0x7f0fd806c650 0x7f0fd80cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.027+0000 7f0fdf200640 1 --2- 172.21.15.163:0/2900564249 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd8159380 0x7f0fd80b2f90 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.027+0000 7f0fdf200640 1 -- 172.21.15.163:0/2900564249 >> 172.21.15.163:0/2900564249 conn(0x7f0fd806d420 msgr2=0x7f0fd806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.027+0000 7f0fdf200640 1 -- 172.21.15.163:0/2900564249 shutdown_connections 2024-07-26T06:17:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.027+0000 7f0fdf200640 1 -- 172.21.15.163:0/2900564249 wait complete. 2024-07-26T06:17:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.029+0000 7f0fdf200640 1 Processor -- start 2024-07-26T06:17:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.029+0000 7f0fdf200640 1 -- start start 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdf200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fd806c650 0x7f0fd8151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdf200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 0x7f0fd813fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdf200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0fd8159380 0x7f0fd81536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdf200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd80ccfa0 con 0x7f0fd80b40a0 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdf200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd80cce20 con 0x7f0fd806c650 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdf200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd80cd120 con 0x7f0fd8159380 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 0x7f0fd813fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 0x7f0fd813fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34034/0 (socket says 172.21.15.163:34034) 2024-07-26T06:17:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fd7400640 1 -- 172.21.15.163:0/3915623826 learned_addr learned my addr 172.21.15.163:0/3915623826 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.030+0000 7f0fdd200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0fd8159380 0x7f0fd81536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7e00640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fd806c650 0x7f0fd8151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7400640 1 -- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0fd8159380 msgr2=0x7f0fd81536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7400640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0fd8159380 0x7f0fd81536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7400640 1 -- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fd806c650 msgr2=0x7f0fd8151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7400640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fd806c650 0x7f0fd8151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7400640 1 -- 172.21.15.163:0/3915623826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd8140050 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fd7400640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 0x7f0fd813fb10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c410 tx=0x7f0fc400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4017070 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fc40035f0 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4003a80 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd8152600 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.031+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd8152bc0 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.032+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd80cc8e0 con 0x7f0fd80b40a0 2024-07-26T06:17:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.034+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f0fc4003c20 con 0x7f0fd80b40a0 2024-07-26T06:17:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.036+0000 7f0fcfe00640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0fb8076660 0x7f0fb8078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.036+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5609+0+0 (secure 0 0 0) 0x7f0fc4097390 con 0x7f0fd80b40a0 2024-07-26T06:17:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.036+0000 7f0fd7e00640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0fb8076660 0x7f0fb8078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:46.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.037+0000 7f0fd7e00640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0fb8076660 0x7f0fb8078b20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8033000 tx=0x7f0fc8004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.039+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0fd8140050 con 0x7f0fd80b40a0 2024-07-26T06:17:46.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.283+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0fd80720a0 con 0x7f0fd80b40a0 2024-07-26T06:17:46.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.846+0000 7f0fcfe00640 1 -- 172.21.15.163:0/3915623826 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) ==== 174+0+0 (secure 0 0 0) 0x7f0fc4060880 con 0x7f0fd80b40a0 2024-07-26T06:17:46.848 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 10 and data pool 11 2024-07-26T06:17:46.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.852+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0fb8076660 msgr2=0x7f0fb8078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:46.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.852+0000 7f0fdf200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0fb8076660 0x7f0fb8078b20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8033000 tx=0x7f0fc8004730 comp rx=0 tx=0).stop 2024-07-26T06:17:46.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 msgr2=0x7f0fd813fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:46.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 0x7f0fd813fb10 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c410 tx=0x7f0fc400c8d0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 shutdown_connections 2024-07-26T06:17:46.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f0fb8076660 0x7f0fb8078b20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0fd8159380 0x7f0fd81536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fd80b40a0 0x7f0fd813fb10 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 --2- 172.21.15.163:0/3915623826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fd806c650 0x7f0fd8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:46.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.853+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 >> 172.21.15.163:0/3915623826 conn(0x7f0fd806d420 msgr2=0x7f0fd80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:46.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.854+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 shutdown_connections 2024-07-26T06:17:46.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:46.854+0000 7f0fdf200640 1 -- 172.21.15.163:0/3915623826 wait complete. 2024-07-26T06:17:46.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:47.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 -- 172.21.15.163:0/3970881530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 msgr2=0x7f54dc161770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 --2- 172.21.15.163:0/3970881530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc161770 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f54dc06e420 tx=0x7f54cc02a5f0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 -- 172.21.15.163:0/3970881530 shutdown_connections 2024-07-26T06:17:47.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 --2- 172.21.15.163:0/3970881530 >> v2:172.21.15.164:3300/0 conn(0x7f54dc0cdd10 0x7f54dc0ce140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 --2- 172.21.15.163:0/3970881530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc161770 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 -- 172.21.15.163:0/3970881530 >> 172.21.15.163:0/3970881530 conn(0x7f54dc06d420 msgr2=0x7f54dc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:47.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 -- 172.21.15.163:0/3970881530 shutdown_connections 2024-07-26T06:17:47.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.053+0000 7f54e1e00640 1 -- 172.21.15.163:0/3970881530 wait complete. 2024-07-26T06:17:47.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.055+0000 7f54e1e00640 1 Processor -- start 2024-07-26T06:17:47.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.055+0000 7f54e1e00640 1 -- start start 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54e1e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54dc06bf40 0x7f54dc13fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54e1e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54dc0cdd10 0x7f54dc159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54e1e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54e1e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc0ce680 con 0x7f54dc1404a0 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54e1e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc161ec0 con 0x7f54dc0cdd10 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54e1e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f54dc0ce800 con 0x7f54dc06bf40 2024-07-26T06:17:47.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54db400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54db400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc072930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34056/0 (socket says 172.21.15.163:34056) 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54db400640 1 -- 172.21.15.163:0/1159376315 learned_addr learned my addr 172.21.15.163:0/1159376315 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54daa00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54dc06bf40 0x7f54dc13fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.056+0000 7f54d2000640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54dc0cdd10 0x7f54dc159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54db400640 1 -- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54dc06bf40 msgr2=0x7f54dc13fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54db400640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54dc06bf40 0x7f54dc13fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54db400640 1 -- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54dc0cdd10 msgr2=0x7f54dc159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54db400640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54dc0cdd10 0x7f54dc159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54db400640 1 -- 172.21.15.163:0/1159376315 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc0b0440 con 0x7f54dc1404a0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54d2000640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54dc0cdd10 0x7f54dc159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54daa00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54dc06bf40 0x7f54dc13fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54db400640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc072930 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f54d400b970 tx=0x7f54d400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d4019070 con 0x7f54dc1404a0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54d400cc50 con 0x7f54dc1404a0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d40146a0 con 0x7f54dc1404a0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc1f4180 con 0x7f54dc1404a0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.057+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc1f4720 con 0x7f54dc1404a0 2024-07-26T06:17:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.058+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54dc1f50c0 con 0x7f54dc1404a0 2024-07-26T06:17:47.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.060+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f54d401c070 con 0x7f54dc1404a0 2024-07-26T06:17:47.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.060+0000 7f54d3e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f54b8076630 0x7f54b8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.061+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f54d4097b10 con 0x7f54dc1404a0 2024-07-26T06:17:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.061+0000 7f54daa00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f54b8076630 0x7f54b8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.062+0000 7f54daa00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f54b8076630 0x7f54b8078af0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f54cc033000 tx=0x7f54cc004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:47.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.065+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f54dc1f50c0 con 0x7f54dc1404a0 2024-07-26T06:17:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.272+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54dc0ced20 con 0x7f54dc1404a0 2024-07-26T06:17:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.274+0000 7f54d3e00640 1 -- 172.21.15.163:0/1159376315 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16705 (secure 0 0 0) 0x7f54dc0ced20 con 0x7f54dc1404a0 2024-07-26T06:17:47.276 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:47.277 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":68,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:46.285885+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:44.680290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:45.272282+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:47.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f54b8076630 msgr2=0x7f54b8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f54b8076630 0x7f54b8078af0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f54cc033000 tx=0x7f54cc004730 comp rx=0 tx=0).stop 2024-07-26T06:17:47.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 msgr2=0x7f54dc072930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc072930 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f54d400b970 tx=0x7f54d400be30 comp rx=0 tx=0).stop 2024-07-26T06:17:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 shutdown_connections 2024-07-26T06:17:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f54b8076630 0x7f54b8078af0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54dc1404a0 0x7f54dc072930 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54dc0cdd10 0x7f54dc159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 --2- 172.21.15.163:0/1159376315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54dc06bf40 0x7f54dc13fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 >> 172.21.15.163:0/1159376315 conn(0x7f54dc06d420 msgr2=0x7f54dc0b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 shutdown_connections 2024-07-26T06:17:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.277+0000 7f54e1e00640 1 -- 172.21.15.163:0/1159376315 wait complete. 2024-07-26T06:17:47.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:47.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0e200640 1 -- 172.21.15.163:0/3930864811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7af0003a80 msgr2=0x7f7af0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3930864811 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7af0003a80 0x7f7af0003ea0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7afc006fb0 tx=0x7f7af0005830 comp rx=0 tx=0).stop 2024-07-26T06:17:47.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0d800640 1 -- 172.21.15.163:0/3930864811 reap_dead start 2024-07-26T06:17:47.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0e200640 1 -- 172.21.15.163:0/3930864811 shutdown_connections 2024-07-26T06:17:47.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0e200640 1 -- 172.21.15.163:0/3930864811 >> 172.21.15.163:0/3930864811 conn(0x7f7b080b0e30 msgr2=0x7f7b080b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:47.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0e200640 1 -- 172.21.15.163:0/3930864811 shutdown_connections 2024-07-26T06:17:47.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.491+0000 7f7b0e200640 1 -- 172.21.15.163:0/3930864811 wait complete. 2024-07-26T06:17:47.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.493+0000 7f7b0e200640 1 Processor -- start 2024-07-26T06:17:47.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.493+0000 7f7b0e200640 1 -- start start 2024-07-26T06:17:47.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0e200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7af0003a80 0x7f7b0814bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0e200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b08073400 0x7f7b08073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0e200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 0x7f7b081564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0e200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080b2d80 con 0x7f7b08156020 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0e200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0806f500 con 0x7f7b08073400 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0e200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7b080b2f00 con 0x7f7af0003a80 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 0x7f7b081564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 0x7f7b081564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34086/0 (socket says 172.21.15.163:34086) 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.494+0000 7f7b0d800640 1 -- 172.21.15.163:0/3004065446 learned_addr learned my addr 172.21.15.163:0/3004065446 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0d800640 1 -- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7af0003a80 msgr2=0x7f7b0814bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0ce00640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7af0003a80 0x7f7b0814bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0d800640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7af0003a80 0x7f7b0814bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b07e00640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b08073400 0x7f7b08073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0d800640 1 -- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b08073400 msgr2=0x7f7b08073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0d800640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b08073400 0x7f7b08073880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0d800640 1 -- 172.21.15.163:0/3004065446 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0814e300 con 0x7f7b08156020 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b07e00640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b08073400 0x7f7b08073880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b0d800640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 0x7f7b081564d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7b080b4340 tx=0x7f7b0000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000f040 con 0x7f7b08156020 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.495+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b00018070 con 0x7f7b08156020 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.496+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b081f1cb0 con 0x7f7b08156020 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.496+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000c900 con 0x7f7b08156020 2024-07-26T06:17:47.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.496+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b081f2230 con 0x7f7b08156020 2024-07-26T06:17:47.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.497+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b080b38c0 con 0x7f7b08156020 2024-07-26T06:17:47.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.498+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f7b0000caa0 con 0x7f7b08156020 2024-07-26T06:17:47.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.499+0000 7f7b04c00640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ae8076630 0x7f7ae8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.499+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f7b000978a0 con 0x7f7b08156020 2024-07-26T06:17:47.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.500+0000 7f7b0ce00640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ae8076630 0x7f7ae8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.501+0000 7f7b0ce00640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ae8076630 0x7f7ae8078af0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7afc033000 tx=0x7f7afc0019a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:47.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.503+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7b080b38c0 con 0x7f7b08156020 2024-07-26T06:17:47.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.692+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b080cd530 con 0x7f7b08156020 2024-07-26T06:17:47.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.695+0000 7f7b04c00640 1 -- 172.21.15.163:0/3004065446 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16705 (secure 0 0 0) 0x7f7b0814e300 con 0x7f7b08156020 2024-07-26T06:17:47.696 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:47.697 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":68,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:46.285885+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:44.680290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:45.272282+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:47.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ae8076630 msgr2=0x7f7ae8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ae8076630 0x7f7ae8078af0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7afc033000 tx=0x7f7afc0019a0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 msgr2=0x7f7b081564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 0x7f7b081564d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7b080b4340 tx=0x7f7b0000ec20 comp rx=0 tx=0).stop 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 shutdown_connections 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7ae8076630 0x7f7ae8078af0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b08156020 0x7f7b081564d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b08073400 0x7f7b08073880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 --2- 172.21.15.163:0/3004065446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7af0003a80 0x7f7b0814bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 >> 172.21.15.163:0/3004065446 conn(0x7f7b080b0e30 msgr2=0x7f7b0815a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:47.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 shutdown_connections 2024-07-26T06:17:47.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.698+0000 7f7b0e200640 1 -- 172.21.15.163:0/3004065446 wait complete. 2024-07-26T06:17:47.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:17:47.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 -- 172.21.15.163:0/998009686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d80b40a0 msgr2=0x7f25d806bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 --2- 172.21.15.163:0/998009686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d80b40a0 0x7f25d806bf40 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f25d8069760 tx=0x7f25c402ade0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 -- 172.21.15.163:0/998009686 shutdown_connections 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 --2- 172.21.15.163:0/998009686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25d806c480 0x7f25d815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 --2- 172.21.15.163:0/998009686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d80b40a0 0x7f25d806bf40 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 --2- 172.21.15.163:0/998009686 >> v2:172.21.15.164:3300/0 conn(0x7f25d8159380 0x7f25d80b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 -- 172.21.15.163:0/998009686 >> 172.21.15.163:0/998009686 conn(0x7f25d806d420 msgr2=0x7f25d806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.933+0000 7f25df400640 1 -- 172.21.15.163:0/998009686 shutdown_connections 2024-07-26T06:17:47.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.934+0000 7f25df400640 1 -- 172.21.15.163:0/998009686 wait complete. 2024-07-26T06:17:47.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.935+0000 7f25df400640 1 Processor -- start 2024-07-26T06:17:47.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.935+0000 7f25df400640 1 -- start start 2024-07-26T06:17:47.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.936+0000 7f25df400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 0x7f25d8147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.936+0000 7f25df400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25d80b40a0 0x7f25d81f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.936+0000 7f25df400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25d8159380 0x7f25d814acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.936+0000 7f25df400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f25d80cdc00 con 0x7f25d806c480 2024-07-26T06:17:47.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.936+0000 7f25df400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f25d80b46b0 con 0x7f25d80b40a0 2024-07-26T06:17:47.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.936+0000 7f25df400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f25d806f370 con 0x7f25d8159380 2024-07-26T06:17:47.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 0x7f25d8147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 0x7f25d8147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34112/0 (socket says 172.21.15.163:34112) 2024-07-26T06:17:47.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 -- 172.21.15.163:0/1166826755 learned_addr learned my addr 172.21.15.163:0/1166826755 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25d7e00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25d80b40a0 0x7f25d81f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 -- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25d8159380 msgr2=0x7f25d814acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25d8159380 0x7f25d814acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 -- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25d80b40a0 msgr2=0x7f25d81f6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25d80b40a0 0x7f25d81f6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 -- 172.21.15.163:0/1166826755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d81f7830 con 0x7f25d806c480 2024-07-26T06:17:47.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.937+0000 7f25dca00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 0x7f25d8147b20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00b440 tx=0x7f25cc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:47.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.938+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc015020 con 0x7f25d806c480 2024-07-26T06:17:47.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.938+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25cc011980 con 0x7f25d806c480 2024-07-26T06:17:47.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.938+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc011dd0 con 0x7f25d806c480 2024-07-26T06:17:47.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.938+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d81f7b10 con 0x7f25d806c480 2024-07-26T06:17:47.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.938+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d8148060 con 0x7f25d806c480 2024-07-26T06:17:47.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.939+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d806bf40 con 0x7f25d806c480 2024-07-26T06:17:47.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.940+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f25cc018020 con 0x7f25d806c480 2024-07-26T06:17:47.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.941+0000 7f25d4c00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25b8076630 0x7f25b8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:47.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.941+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f25cc097e00 con 0x7f25d806c480 2024-07-26T06:17:47.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.942+0000 7f25d7e00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25b8076630 0x7f25b8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:47.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.943+0000 7f25d7e00640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25b8076630 0x7f25b8078af0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f25c4034000 tx=0x7f25c4004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:47.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:47.945+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f25cc01f020 con 0x7f25d806c480 2024-07-26T06:17:48.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.176+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f25d8069050 con 0x7f25d806c480 2024-07-26T06:17:48.752 INFO:tasks.ceph.mds.c.smithi163.stderr:2024-07-26T06:17:48.751+0000 7fa392600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:17:48.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.875+0000 7f25d4c00640 1 -- 172.21.15.163:0/1166826755 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) ==== 118+0+0 (secure 0 0 0) 0x7f25cc00d040 con 0x7f25d806c480 2024-07-26T06:17:48.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25b8076630 msgr2=0x7f25b8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:48.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25b8076630 0x7f25b8078af0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f25c4034000 tx=0x7f25c4004730 comp rx=0 tx=0).stop 2024-07-26T06:17:48.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 msgr2=0x7f25d8147b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:48.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 0x7f25d8147b20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00b440 tx=0x7f25cc00b900 comp rx=0 tx=0).stop 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 shutdown_connections 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f25b8076630 0x7f25b8078af0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25d8159380 0x7f25d814acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25d80b40a0 0x7f25d81f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 --2- 172.21.15.163:0/1166826755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d806c480 0x7f25d8147b20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.879+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 >> 172.21.15.163:0/1166826755 conn(0x7f25d806d420 msgr2=0x7f25d80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.880+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 shutdown_connections 2024-07-26T06:17:48.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:48.880+0000 7f25df400640 1 -- 172.21.15.163:0/1166826755 wait complete. 2024-07-26T06:17:48.904 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:17:48.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:17:49.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 -- 172.21.15.163:0/830730142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080b1e80 msgr2=0x7f98080b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:49.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 --2- 172.21.15.163:0/830730142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080b1e80 0x7f98080b6bd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9800005fd0 tx=0x7f980002a8e0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 -- 172.21.15.163:0/830730142 shutdown_connections 2024-07-26T06:17:49.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 --2- 172.21.15.163:0/830730142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080b1e80 0x7f98080b6bd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 --2- 172.21.15.163:0/830730142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f980815d5a0 0x7f98080ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 -- 172.21.15.163:0/830730142 >> 172.21.15.163:0/830730142 conn(0x7f98080cfda0 msgr2=0x7f98080d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:49.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 -- 172.21.15.163:0/830730142 shutdown_connections 2024-07-26T06:17:49.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.115+0000 7f980e400640 1 -- 172.21.15.163:0/830730142 wait complete. 2024-07-26T06:17:49.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.117+0000 7f980e400640 1 Processor -- start 2024-07-26T06:17:49.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.117+0000 7f980e400640 1 -- start start 2024-07-26T06:17:49.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f980e400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98080b1e80 0x7f98081602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f980e400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 0x7f98080ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f980e400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f980815d5a0 0x7f98080c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f980e400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f98080b5130 con 0x7f98080ce830 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f980e400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f98080b4fb0 con 0x7f980815d5a0 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f980e400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f98080b52b0 con 0x7f98080b1e80 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f9807e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 0x7f98080ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f9807e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 0x7f98080ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34134/0 (socket says 172.21.15.163:34134) 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.118+0000 7f9807e00640 1 -- 172.21.15.163:0/52744772 learned_addr learned my addr 172.21.15.163:0/52744772 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f980d000640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98080b1e80 0x7f98081602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9807e00640 1 -- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98080b1e80 msgr2=0x7f98081602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f980da00640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f980815d5a0 0x7f98080c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9807e00640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98080b1e80 0x7f98081602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9807e00640 1 -- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f980815d5a0 msgr2=0x7f98080c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:49.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9807e00640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f980815d5a0 0x7f98080c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9807e00640 1 -- 172.21.15.163:0/52744772 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98081607e0 con 0x7f98080ce830 2024-07-26T06:17:49.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9807e00640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 0x7f98080ccaa0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c400 tx=0x7f97f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:49.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0011790 con 0x7f98080ce830 2024-07-26T06:17:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97f0017070 con 0x7f98080ce830 2024-07-26T06:17:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0011d00 con 0x7f98080ce830 2024-07-26T06:17:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.119+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98081f3af0 con 0x7f98080ce830 2024-07-26T06:17:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.120+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98081f3d90 con 0x7f98080ce830 2024-07-26T06:17:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.121+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98080ce2f0 con 0x7f98080ce830 2024-07-26T06:17:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.122+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f97f001c460 con 0x7f98080ce830 2024-07-26T06:17:49.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.123+0000 7f9804c00640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f97e4076630 0x7f97e4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:49.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.124+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f97f0097c00 con 0x7f98080ce830 2024-07-26T06:17:49.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.124+0000 7f980d000640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f97e4076630 0x7f97e4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:49.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.125+0000 7f980d000640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f97e4076630 0x7f97e4078af0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f97f8005fd0 tx=0x7f97f8005d80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:49.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.128+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97f00611a0 con 0x7f98080ce830 2024-07-26T06:17:49.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.378+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f98080b3d00 con 0x7f98080ce830 2024-07-26T06:17:49.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.886+0000 7f9804c00640 1 -- 172.21.15.163:0/52744772 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) ==== 122+0+0 (secure 0 0 0) 0x7f97f0069190 con 0x7f98080ce830 2024-07-26T06:17:49.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f97e4076630 msgr2=0x7f97e4078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:49.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f97e4076630 0x7f97e4078af0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f97f8005fd0 tx=0x7f97f8005d80 comp rx=0 tx=0).stop 2024-07-26T06:17:49.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 msgr2=0x7f98080ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:49.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 0x7f98080ccaa0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c400 tx=0x7f97f000c8c0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 shutdown_connections 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f97e4076630 0x7f97e4078af0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f980815d5a0 0x7f98080c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98080ce830 0x7f98080ccaa0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 --2- 172.21.15.163:0/52744772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98080b1e80 0x7f98081602a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 >> 172.21.15.163:0/52744772 conn(0x7f98080cfda0 msgr2=0x7f98080b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 shutdown_connections 2024-07-26T06:17:49.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:49.891+0000 7f980e400640 1 -- 172.21.15.163:0/52744772 wait complete. 2024-07-26T06:17:49.915 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:17:49.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 -- 172.21.15.163:0/1676780154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b23d0 msgr2=0x7fe1780b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 --2- 172.21.15.163:0/1676780154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b23d0 0x7fe1780b6bd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe170005fd0 tx=0x7fe17002a980 comp rx=0 tx=0).stop 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 -- 172.21.15.163:0/1676780154 shutdown_connections 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 --2- 172.21.15.163:0/1676780154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b23d0 0x7fe1780b6bd0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 -- 172.21.15.163:0/1676780154 >> 172.21.15.163:0/1676780154 conn(0x7fe1780cfda0 msgr2=0x7fe1780d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 -- 172.21.15.163:0/1676780154 shutdown_connections 2024-07-26T06:17:50.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.114+0000 7fe17ec00640 1 -- 172.21.15.163:0/1676780154 wait complete. 2024-07-26T06:17:50.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.116+0000 7fe17ec00640 1 Processor -- start 2024-07-26T06:17:50.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.116+0000 7fe17ec00640 1 -- start start 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ec00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 0x7fe17815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ec00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1780b23d0 0x7fe1781598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ec00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17815d5a0 0x7fe178159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ec00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe1780b5130 con 0x7fe1780b1e80 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ec00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe1780b4fb0 con 0x7fe1780b23d0 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ec00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe1780b52b0 con 0x7fe17815d5a0 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 0x7fe17815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 0x7fe17815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34148/0 (socket says 172.21.15.163:34148) 2024-07-26T06:17:50.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17d800640 1 -- 172.21.15.163:0/2532586741 learned_addr learned my addr 172.21.15.163:0/2532586741 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:50.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17ce00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1780b23d0 0x7fe1781598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.117+0000 7fe17e200640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17815d5a0 0x7fe178159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17d800640 1 -- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17815d5a0 msgr2=0x7fe178159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17d800640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17815d5a0 0x7fe178159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17d800640 1 -- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1780b23d0 msgr2=0x7fe1781598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17d800640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1780b23d0 0x7fe1781598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17d800640 1 -- 172.21.15.163:0/2532586741 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe178160c20 con 0x7fe1780b1e80 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17ce00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1780b23d0 0x7fe1781598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17d800640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 0x7fe17815f810 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00b420 tx=0x7fe16c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c011830 con 0x7fe1780b1e80 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe16c016070 con 0x7fe1780b1e80 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c011d60 con 0x7fe1780b1e80 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1781600a0 con 0x7fe1780b1e80 2024-07-26T06:17:50.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.118+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe178160660 con 0x7fe1780b1e80 2024-07-26T06:17:50.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.120+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1780b57a0 con 0x7fe1780b1e80 2024-07-26T06:17:50.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.121+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fe16c01b460 con 0x7fe1780b1e80 2024-07-26T06:17:50.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.122+0000 7fe175600640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe158076630 0x7fe158078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.122+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fe16c097840 con 0x7fe1780b1e80 2024-07-26T06:17:50.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.122+0000 7fe17ce00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe158076630 0x7fe158078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.123+0000 7fe17ce00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe158076630 0x7fe158078af0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe17815a9b0 tx=0x7fe164009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:50.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.126+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe16c020020 con 0x7fe1780b1e80 2024-07-26T06:17:50.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.375+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1780d0320 con 0x7fe1780b1e80 2024-07-26T06:17:50.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.376+0000 7fe175600640 1 -- 172.21.15.163:0/2532586741 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7fe16c060d50 con 0x7fe1780b1e80 2024-07-26T06:17:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:50.377 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":45,"btime":"2024-07-26T06:17:49:877600+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:17:50.377 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 45 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe158076630 msgr2=0x7fe158078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe158076630 0x7fe158078af0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe17815a9b0 tx=0x7fe164009040 comp rx=0 tx=0).stop 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 msgr2=0x7fe17815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 0x7fe17815f810 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00b420 tx=0x7fe16c00b8e0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 shutdown_connections 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe158076630 0x7fe158078af0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17815d5a0 0x7fe178159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1780b23d0 0x7fe1781598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 --2- 172.21.15.163:0/2532586741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1780b1e80 0x7fe17815f810 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 >> 172.21.15.163:0/2532586741 conn(0x7fe1780cfda0 msgr2=0x7fe17815aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:50.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 shutdown_connections 2024-07-26T06:17:50.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.379+0000 7fe17ec00640 1 -- 172.21.15.163:0/2532586741 wait complete. 2024-07-26T06:17:50.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.603+0000 7f6f6cc00640 1 -- 172.21.15.163:0/1017534913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f681556f0 msgr2=0x7f6f680cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.603+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/1017534913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f681556f0 0x7f6f680cdd90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f58005fd0 tx=0x7f6f5802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.603+0000 7f6f6cc00640 1 -- 172.21.15.163:0/1017534913 shutdown_connections 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.604+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/1017534913 >> v2:172.21.15.164:3300/0 conn(0x7f6f68071c40 0x7f6f680b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.604+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/1017534913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f681556f0 0x7f6f680cdd90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.604+0000 7f6f6cc00640 1 -- 172.21.15.163:0/1017534913 >> 172.21.15.163:0/1017534913 conn(0x7f6f680b0e30 msgr2=0x7f6f680b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.604+0000 7f6f6cc00640 1 -- 172.21.15.163:0/1017534913 shutdown_connections 2024-07-26T06:17:50.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.604+0000 7f6f6cc00640 1 -- 172.21.15.163:0/1017534913 wait complete. 2024-07-26T06:17:50.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.605+0000 7f6f6cc00640 1 Processor -- start 2024-07-26T06:17:50.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.605+0000 7f6f6cc00640 1 -- start start 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.606+0000 7f6f6cc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f68071c40 0x7f6f6814fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.606+0000 7f6f6cc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f680ce2d0 0x7f6f6815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.606+0000 7f6f6cc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 0x7f6f680a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.606+0000 7f6f6cc00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6f680b4020 con 0x7f6f681556f0 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.606+0000 7f6f6cc00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6f680b3ea0 con 0x7f6f68071c40 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.606+0000 7f6f6cc00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6f680b41a0 con 0x7f6f680ce2d0 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 0x7f6f680a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 0x7f6f680a94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34164/0 (socket says 172.21.15.163:34164) 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 -- 172.21.15.163:0/3229616723 learned_addr learned my addr 172.21.15.163:0/3229616723 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f5ea00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f680ce2d0 0x7f6f6815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67400640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f68071c40 0x7f6f6814fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 -- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f680ce2d0 msgr2=0x7f6f6815a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f680ce2d0 0x7f6f6815a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 -- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f68071c40 msgr2=0x7f6f6814fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f68071c40 0x7f6f6814fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 -- 172.21.15.163:0/3229616723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f680a9f70 con 0x7f6f681556f0 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67400640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f68071c40 0x7f6f6814fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:50.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.607+0000 7f6f67e00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 0x7f6f680a94d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000ead0 tx=0x7f6f6000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:50.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.608+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6000cde0 con 0x7f6f681556f0 2024-07-26T06:17:50.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.608+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f680aa250 con 0x7f6f681556f0 2024-07-26T06:17:50.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.608+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f60004c10 con 0x7f6f681556f0 2024-07-26T06:17:50.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.608+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60005020 con 0x7f6f681556f0 2024-07-26T06:17:50.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.608+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6815a080 con 0x7f6f681556f0 2024-07-26T06:17:50.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.609+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f680b46c0 con 0x7f6f681556f0 2024-07-26T06:17:50.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.610+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f6f600051c0 con 0x7f6f681556f0 2024-07-26T06:17:50.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.611+0000 7f6f5fe00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6f44076630 0x7f6f44078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:50.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.611+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f6f60098810 con 0x7f6f681556f0 2024-07-26T06:17:50.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.612+0000 7f6f67400640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6f44076630 0x7f6f44078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:50.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.613+0000 7f6f67400640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6f44076630 0x7f6f44078af0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6f58033000 tx=0x7f6f58004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:50.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.615+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f60016030 con 0x7f6f681556f0 2024-07-26T06:17:50.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.819+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f680b34f0 con 0x7f6f681556f0 2024-07-26T06:17:50.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.822+0000 7f6f5fe00640 1 -- 172.21.15.163:0/3229616723 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16692 (secure 0 0 0) 0x7f6f60061b20 con 0x7f6f681556f0 2024-07-26T06:17:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:50.824 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":69,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:17:47.846992+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:44.680290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:45.272282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:17:50.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6f44076630 msgr2=0x7f6f44078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6f44076630 0x7f6f44078af0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6f58033000 tx=0x7f6f58004730 comp rx=0 tx=0).stop 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 msgr2=0x7f6f680a94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 0x7f6f680a94d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000ead0 tx=0x7f6f6000ef90 comp rx=0 tx=0).stop 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 shutdown_connections 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6f44076630 0x7f6f44078af0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f681556f0 0x7f6f680a94d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f680ce2d0 0x7f6f6815a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 --2- 172.21.15.163:0/3229616723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f68071c40 0x7f6f6814fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.825+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 >> 172.21.15.163:0/3229616723 conn(0x7f6f680b0e30 msgr2=0x7f6f681520e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.826+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 shutdown_connections 2024-07-26T06:17:50.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:50.826+0000 7f6f6cc00640 1 -- 172.21.15.163:0/3229616723 wait complete. 2024-07-26T06:17:50.847 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:17:50.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.042+0000 7fc452000640 1 -- 172.21.15.163:0/1049207442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc434003a00 msgr2=0x7fc434003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.042+0000 7fc452000640 1 --2- 172.21.15.163:0/1049207442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc434003a00 0x7fc434003e20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc44000c2e0 tx=0x7fc434005830 comp rx=0 tx=0).stop 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.042+0000 7fc451600640 1 -- 172.21.15.163:0/1049207442 reap_dead start 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.043+0000 7fc452000640 1 -- 172.21.15.163:0/1049207442 shutdown_connections 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.043+0000 7fc452000640 1 -- 172.21.15.163:0/1049207442 >> 172.21.15.163:0/1049207442 conn(0x7fc44c0b0e30 msgr2=0x7fc44c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.043+0000 7fc452000640 1 -- 172.21.15.163:0/1049207442 shutdown_connections 2024-07-26T06:17:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.043+0000 7fc452000640 1 -- 172.21.15.163:0/1049207442 wait complete. 2024-07-26T06:17:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.044+0000 7fc452000640 1 Processor -- start 2024-07-26T06:17:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.044+0000 7fc452000640 1 -- start start 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.045+0000 7fc452000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc434003a00 0x7fc44c143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.045+0000 7fc452000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 0x7fc44c153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.045+0000 7fc452000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc44c0b0090 0x7fc44c0b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.045+0000 7fc452000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c0cb9f0 con 0x7fc44c0caa00 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.045+0000 7fc452000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c0cb870 con 0x7fc434003a00 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.045+0000 7fc452000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c15a1d0 con 0x7fc44c0b0090 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 0x7fc44c153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 0x7fc44c153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34180/0 (socket says 172.21.15.163:34180) 2024-07-26T06:17:51.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 -- 172.21.15.163:0/1467508994 learned_addr learned my addr 172.21.15.163:0/1467508994 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc450c00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc434003a00 0x7fc44c143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 -- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc44c0b0090 msgr2=0x7fc44c0b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc44c0b0090 0x7fc44c0b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 -- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc434003a00 msgr2=0x7fc44c143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc434003a00 0x7fc44c143670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 -- 172.21.15.163:0/1467508994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c1538c0 con 0x7fc44c0caa00 2024-07-26T06:17:51.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.046+0000 7fc44be00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 0x7fc44c153380 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc43800d950 tx=0x7fc43800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.047+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438009c30 con 0x7fc44c0caa00 2024-07-26T06:17:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.047+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc438004030 con 0x7fc44c0caa00 2024-07-26T06:17:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.047+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438004480 con 0x7fc44c0caa00 2024-07-26T06:17:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.047+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44c143d30 con 0x7fc44c0caa00 2024-07-26T06:17:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.047+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44c1442f0 con 0x7fc44c0caa00 2024-07-26T06:17:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.048+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc44c0b26d0 con 0x7fc44c0caa00 2024-07-26T06:17:51.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.049+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fc4380035f0 con 0x7fc44c0caa00 2024-07-26T06:17:51.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.050+0000 7fc448c00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc42c076630 0x7fc42c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.050+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fc438098330 con 0x7fc44c0caa00 2024-07-26T06:17:51.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.051+0000 7fc450c00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc42c076630 0x7fc42c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:51.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.052+0000 7fc450c00640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc42c076630 0x7fc42c078af0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc440033000 tx=0x7fc44002d000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:51.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.054+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc438016020 con 0x7fc44c0caa00 2024-07-26T06:17:51.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.256+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc44c0cc4b0 con 0x7fc42c076630 2024-07-26T06:17:51.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.269+0000 7fc448c00640 1 -- 172.21.15.163:0/1467508994 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250333 (secure 0 0 0) 0x7fc44c0cc4b0 con 0x7fc42c076630 2024-07-26T06:17:51.274 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:51.276 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:17:51.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.274+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc42c076630 msgr2=0x7fc42c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.274+0000 7fc452000640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc42c076630 0x7fc42c078af0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc440033000 tx=0x7fc44002d000 comp rx=0 tx=0).stop 2024-07-26T06:17:51.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.274+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 msgr2=0x7fc44c153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.274+0000 7fc452000640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 0x7fc44c153380 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc43800d950 tx=0x7fc43800de10 comp rx=0 tx=0).stop 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 shutdown_connections 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fc42c076630 0x7fc42c078af0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc44c0b0090 0x7fc44c0b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc44c0caa00 0x7fc44c153380 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 --2- 172.21.15.163:0/1467508994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc434003a00 0x7fc44c143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 >> 172.21.15.163:0/1467508994 conn(0x7fc44c0b0e30 msgr2=0x7fc44c0b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 shutdown_connections 2024-07-26T06:17:51.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.275+0000 7fc452000640 1 -- 172.21.15.163:0/1467508994 wait complete. 2024-07-26T06:17:51.294 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-07-26T06:17:49.742156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591042,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":114,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252820,"kb_used_data":7512,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721900,"statfs":{"total":773094113280,"available":772835225600,"internally_reserved":0,"allocated":7692288,"data_stored":3887044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":674,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.942297"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.877704+0000","last_change":"2024-07-26T06:17:46.868114+0000","last_active":"2024-07-26T06:17:46.877704+0000","last_peered":"2024-07-26T06:17:46.877704+0000","last_clean":"2024-07-26T06:17:46.877704+0000","last_became_active":"2024-07-26T06:17:46.867785+0000","last_became_peered":"2024-07-26T06:17:46.867785+0000","last_unstale":"2024-07-26T06:17:46.877704+0000","last_undegraded":"2024-07-26T06:17:46.877704+0000","last_fullsized":"2024-07-26T06:17:46.877704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:06.947898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.878455+0000","last_change":"2024-07-26T06:17:46.872554+0000","last_active":"2024-07-26T06:17:46.878455+0000","last_peered":"2024-07-26T06:17:46.878455+0000","last_clean":"2024-07-26T06:17:46.878455+0000","last_became_active":"2024-07-26T06:17:46.871064+0000","last_became_peered":"2024-07-26T06:17:46.871064+0000","last_unstale":"2024-07-26T06:17:46.878455+0000","last_undegraded":"2024-07-26T06:17:46.878455+0000","last_fullsized":"2024-07-26T06:17:46.878455+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:28.285738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.812841+0000","last_change":"2024-07-26T06:17:45.812841+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.812841+0000","last_undegraded":"2024-07-26T06:17:45.812841+0000","last_fullsized":"2024-07-26T06:17:45.812841+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.826878+0000","last_change":"2024-07-26T06:17:45.826878+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.826878+0000","last_undegraded":"2024-07-26T06:17:45.826878+0000","last_fullsized":"2024-07-26T06:17:45.826878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880634+0000","last_change":"2024-07-26T06:17:46.872713+0000","last_active":"2024-07-26T06:17:46.880634+0000","last_peered":"2024-07-26T06:17:46.880634+0000","last_clean":"2024-07-26T06:17:46.880634+0000","last_became_active":"2024-07-26T06:17:46.871978+0000","last_became_peered":"2024-07-26T06:17:46.871978+0000","last_unstale":"2024-07-26T06:17:46.880634+0000","last_undegraded":"2024-07-26T06:17:46.880634+0000","last_fullsized":"2024-07-26T06:17:46.880634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:30.620660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886672+0000","last_change":"2024-07-26T06:17:46.880828+0000","last_active":"2024-07-26T06:17:46.886672+0000","last_peered":"2024-07-26T06:17:46.886672+0000","last_clean":"2024-07-26T06:17:46.886672+0000","last_became_active":"2024-07-26T06:17:46.875622+0000","last_became_peered":"2024-07-26T06:17:46.875622+0000","last_unstale":"2024-07-26T06:17:46.886672+0000","last_undegraded":"2024-07-26T06:17:46.886672+0000","last_fullsized":"2024-07-26T06:17:46.886672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:08.561347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869796+0000","last_change":"2024-07-26T06:17:46.865079+0000","last_active":"2024-07-26T06:17:46.869796+0000","last_peered":"2024-07-26T06:17:46.869796+0000","last_clean":"2024-07-26T06:17:46.869796+0000","last_became_active":"2024-07-26T06:17:46.864008+0000","last_became_peered":"2024-07-26T06:17:46.864008+0000","last_unstale":"2024-07-26T06:17:46.869796+0000","last_undegraded":"2024-07-26T06:17:46.869796+0000","last_fullsized":"2024-07-26T06:17:46.869796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:35.804674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.881946+0000","last_change":"2024-07-26T06:17:46.871894+0000","last_active":"2024-07-26T06:17:46.881946+0000","last_peered":"2024-07-26T06:17:46.881946+0000","last_clean":"2024-07-26T06:17:46.881946+0000","last_became_active":"2024-07-26T06:17:46.870785+0000","last_became_peered":"2024-07-26T06:17:46.870785+0000","last_unstale":"2024-07-26T06:17:46.881946+0000","last_undegraded":"2024-07-26T06:17:46.881946+0000","last_fullsized":"2024-07-26T06:17:46.881946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:04:18.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880646+0000","last_change":"2024-07-26T06:17:46.872337+0000","last_active":"2024-07-26T06:17:46.880646+0000","last_peered":"2024-07-26T06:17:46.880646+0000","last_clean":"2024-07-26T06:17:46.880646+0000","last_became_active":"2024-07-26T06:17:46.872037+0000","last_became_peered":"2024-07-26T06:17:46.872037+0000","last_unstale":"2024-07-26T06:17:46.880646+0000","last_undegraded":"2024-07-26T06:17:46.880646+0000","last_fullsized":"2024-07-26T06:17:46.880646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:18:52.079504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.818191+0000","last_change":"2024-07-26T06:17:45.818191+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.818191+0000","last_undegraded":"2024-07-26T06:17:45.818191+0000","last_fullsized":"2024-07-26T06:17:45.818191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886860+0000","last_change":"2024-07-26T06:17:46.874454+0000","last_active":"2024-07-26T06:17:46.886860+0000","last_peered":"2024-07-26T06:17:46.886860+0000","last_clean":"2024-07-26T06:17:46.886860+0000","last_became_active":"2024-07-26T06:17:46.874142+0000","last_became_peered":"2024-07-26T06:17:46.874142+0000","last_unstale":"2024-07-26T06:17:46.886860+0000","last_undegraded":"2024-07-26T06:17:46.886860+0000","last_fullsized":"2024-07-26T06:17:46.886860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:37:09.636337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.826375+0000","last_change":"2024-07-26T06:17:45.826375+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.826375+0000","last_undegraded":"2024-07-26T06:17:45.826375+0000","last_fullsized":"2024-07-26T06:17:45.826375+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880811+0000","last_change":"2024-07-26T06:17:46.877319+0000","last_active":"2024-07-26T06:17:46.880811+0000","last_peered":"2024-07-26T06:17:46.880811+0000","last_clean":"2024-07-26T06:17:46.880811+0000","last_became_active":"2024-07-26T06:17:46.877008+0000","last_became_peered":"2024-07-26T06:17:46.877008+0000","last_unstale":"2024-07-26T06:17:46.880811+0000","last_undegraded":"2024-07-26T06:17:46.880811+0000","last_fullsized":"2024-07-26T06:17:46.880811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:46.672362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869852+0000","last_change":"2024-07-26T06:17:46.865837+0000","last_active":"2024-07-26T06:17:46.869852+0000","last_peered":"2024-07-26T06:17:46.869852+0000","last_clean":"2024-07-26T06:17:46.869852+0000","last_became_active":"2024-07-26T06:17:46.865510+0000","last_became_peered":"2024-07-26T06:17:46.865510+0000","last_unstale":"2024-07-26T06:17:46.869852+0000","last_undegraded":"2024-07-26T06:17:46.869852+0000","last_fullsized":"2024-07-26T06:17:46.869852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:54.501776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.863933+0000","last_change":"2024-07-26T06:17:46.857332+0000","last_active":"2024-07-26T06:17:46.863933+0000","last_peered":"2024-07-26T06:17:46.863933+0000","last_clean":"2024-07-26T06:17:46.863933+0000","last_became_active":"2024-07-26T06:17:46.857021+0000","last_became_peered":"2024-07-26T06:17:46.857021+0000","last_unstale":"2024-07-26T06:17:46.863933+0000","last_undegraded":"2024-07-26T06:17:46.863933+0000","last_fullsized":"2024-07-26T06:17:46.863933+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:28.166488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.815234+0000","last_change":"2024-07-26T06:17:45.815234+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.815234+0000","last_undegraded":"2024-07-26T06:17:45.815234+0000","last_fullsized":"2024-07-26T06:17:45.815234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880665+0000","last_change":"2024-07-26T06:17:46.872692+0000","last_active":"2024-07-26T06:17:46.880665+0000","last_peered":"2024-07-26T06:17:46.880665+0000","last_clean":"2024-07-26T06:17:46.880665+0000","last_became_active":"2024-07-26T06:17:46.872397+0000","last_became_peered":"2024-07-26T06:17:46.872397+0000","last_unstale":"2024-07-26T06:17:46.880665+0000","last_undegraded":"2024-07-26T06:17:46.880665+0000","last_fullsized":"2024-07-26T06:17:46.880665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:54.857970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.826604+0000","last_change":"2024-07-26T06:17:45.826604+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.826604+0000","last_undegraded":"2024-07-26T06:17:45.826604+0000","last_fullsized":"2024-07-26T06:17:45.826604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886904+0000","last_change":"2024-07-26T06:17:46.876973+0000","last_active":"2024-07-26T06:17:46.886904+0000","last_peered":"2024-07-26T06:17:46.886904+0000","last_clean":"2024-07-26T06:17:46.886904+0000","last_became_active":"2024-07-26T06:17:46.876656+0000","last_became_peered":"2024-07-26T06:17:46.876656+0000","last_unstale":"2024-07-26T06:17:46.886904+0000","last_undegraded":"2024-07-26T06:17:46.886904+0000","last_fullsized":"2024-07-26T06:17:46.886904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:34.566927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.871423+0000","last_change":"2024-07-26T06:17:46.864998+0000","last_active":"2024-07-26T06:17:46.871423+0000","last_peered":"2024-07-26T06:17:46.871423+0000","last_clean":"2024-07-26T06:17:46.871423+0000","last_became_active":"2024-07-26T06:17:46.864705+0000","last_became_peered":"2024-07-26T06:17:46.864705+0000","last_unstale":"2024-07-26T06:17:46.871423+0000","last_undegraded":"2024-07-26T06:17:46.871423+0000","last_fullsized":"2024-07-26T06:17:46.871423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:38:27.710065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.820189+0000","last_change":"2024-07-26T06:17:45.820189+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.820189+0000","last_undegraded":"2024-07-26T06:17:45.820189+0000","last_fullsized":"2024-07-26T06:17:45.820189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.817633+0000","last_change":"2024-07-26T06:17:45.817633+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.817633+0000","last_undegraded":"2024-07-26T06:17:45.817633+0000","last_fullsized":"2024-07-26T06:17:45.817633+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.883610+0000","last_change":"2024-07-26T06:17:46.872123+0000","last_active":"2024-07-26T06:17:46.883610+0000","last_peered":"2024-07-26T06:17:46.883610+0000","last_clean":"2024-07-26T06:17:46.883610+0000","last_became_active":"2024-07-26T06:17:46.871778+0000","last_became_peered":"2024-07-26T06:17:46.871778+0000","last_unstale":"2024-07-26T06:17:46.883610+0000","last_undegraded":"2024-07-26T06:17:46.883610+0000","last_fullsized":"2024-07-26T06:17:46.883610+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:28.360976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887063+0000","last_change":"2024-07-26T06:17:46.879628+0000","last_active":"2024-07-26T06:17:46.887063+0000","last_peered":"2024-07-26T06:17:46.887063+0000","last_clean":"2024-07-26T06:17:46.887063+0000","last_became_active":"2024-07-26T06:17:46.879345+0000","last_became_peered":"2024-07-26T06:17:46.879345+0000","last_unstale":"2024-07-26T06:17:46.887063+0000","last_undegraded":"2024-07-26T06:17:46.887063+0000","last_fullsized":"2024-07-26T06:17:46.887063+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:34.735640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.817460+0000","last_change":"2024-07-26T06:17:45.817460+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.817460+0000","last_undegraded":"2024-07-26T06:17:45.817460+0000","last_fullsized":"2024-07-26T06:17:45.817460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886724+0000","last_change":"2024-07-26T06:17:46.884327+0000","last_active":"2024-07-26T06:17:46.886724+0000","last_peered":"2024-07-26T06:17:46.886724+0000","last_clean":"2024-07-26T06:17:46.886724+0000","last_became_active":"2024-07-26T06:17:46.882222+0000","last_became_peered":"2024-07-26T06:17:46.882222+0000","last_unstale":"2024-07-26T06:17:46.886724+0000","last_undegraded":"2024-07-26T06:17:46.886724+0000","last_fullsized":"2024-07-26T06:17:46.886724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:32.260066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.812049+0000","last_change":"2024-07-26T06:17:45.812049+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.812049+0000","last_undegraded":"2024-07-26T06:17:45.812049+0000","last_fullsized":"2024-07-26T06:17:45.812049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.816129+0000","last_change":"2024-07-26T06:17:45.816129+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.816129+0000","last_undegraded":"2024-07-26T06:17:45.816129+0000","last_fullsized":"2024-07-26T06:17:45.816129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.812839+0000","last_change":"2024-07-26T06:17:45.812839+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.812839+0000","last_undegraded":"2024-07-26T06:17:45.812839+0000","last_fullsized":"2024-07-26T06:17:45.812839+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886875+0000","last_change":"2024-07-26T06:17:46.883094+0000","last_active":"2024-07-26T06:17:46.886875+0000","last_peered":"2024-07-26T06:17:46.886875+0000","last_clean":"2024-07-26T06:17:46.886875+0000","last_became_active":"2024-07-26T06:17:46.882798+0000","last_became_peered":"2024-07-26T06:17:46.882798+0000","last_unstale":"2024-07-26T06:17:46.886875+0000","last_undegraded":"2024-07-26T06:17:46.886875+0000","last_fullsized":"2024-07-26T06:17:46.886875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:28.360976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869019+0000","last_change":"2024-07-26T06:17:46.864323+0000","last_active":"2024-07-26T06:17:46.869019+0000","last_peered":"2024-07-26T06:17:46.869019+0000","last_clean":"2024-07-26T06:17:46.869019+0000","last_became_active":"2024-07-26T06:17:46.863716+0000","last_became_peered":"2024-07-26T06:17:46.863716+0000","last_unstale":"2024-07-26T06:17:46.869019+0000","last_undegraded":"2024-07-26T06:17:46.869019+0000","last_fullsized":"2024-07-26T06:17:46.869019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:38:27.710065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.881926+0000","last_change":"2024-07-26T06:17:46.871077+0000","last_active":"2024-07-26T06:17:46.881926+0000","last_peered":"2024-07-26T06:17:46.881926+0000","last_clean":"2024-07-26T06:17:46.881926+0000","last_became_active":"2024-07-26T06:17:46.869322+0000","last_became_peered":"2024-07-26T06:17:46.869322+0000","last_unstale":"2024-07-26T06:17:46.881926+0000","last_undegraded":"2024-07-26T06:17:46.881926+0000","last_fullsized":"2024-07-26T06:17:46.881926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:28.015264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887161+0000","last_change":"2024-07-26T06:17:46.881105+0000","last_active":"2024-07-26T06:17:46.887161+0000","last_peered":"2024-07-26T06:17:46.887161+0000","last_clean":"2024-07-26T06:17:46.887161+0000","last_became_active":"2024-07-26T06:17:46.880809+0000","last_became_peered":"2024-07-26T06:17:46.880809+0000","last_unstale":"2024-07-26T06:17:46.887161+0000","last_undegraded":"2024-07-26T06:17:46.887161+0000","last_fullsized":"2024-07-26T06:17:46.887161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:41:51.520946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886637+0000","last_change":"2024-07-26T06:17:45.848547+0000","last_active":"2024-07-26T06:17:46.886637+0000","last_peered":"2024-07-26T06:17:46.886637+0000","last_clean":"2024-07-26T06:17:46.886637+0000","last_became_active":"2024-07-26T06:17:45.848169+0000","last_became_peered":"2024-07-26T06:17:45.848169+0000","last_unstale":"2024-07-26T06:17:46.886637+0000","last_undegraded":"2024-07-26T06:17:46.886637+0000","last_fullsized":"2024-07-26T06:17:46.886637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:34.260020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880622+0000","last_change":"2024-07-26T06:17:46.871455+0000","last_active":"2024-07-26T06:17:46.880622+0000","last_peered":"2024-07-26T06:17:46.880622+0000","last_clean":"2024-07-26T06:17:46.880622+0000","last_became_active":"2024-07-26T06:17:46.871207+0000","last_became_peered":"2024-07-26T06:17:46.871207+0000","last_unstale":"2024-07-26T06:17:46.880622+0000","last_undegraded":"2024-07-26T06:17:46.880622+0000","last_fullsized":"2024-07-26T06:17:46.880622+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:46.694674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880977+0000","last_change":"2024-07-26T06:17:45.831881+0000","last_active":"2024-07-26T06:17:46.880977+0000","last_peered":"2024-07-26T06:17:46.880977+0000","last_clean":"2024-07-26T06:17:46.880977+0000","last_became_active":"2024-07-26T06:17:45.831631+0000","last_became_peered":"2024-07-26T06:17:45.831631+0000","last_unstale":"2024-07-26T06:17:46.880977+0000","last_undegraded":"2024-07-26T06:17:46.880977+0000","last_fullsized":"2024-07-26T06:17:46.880977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:18.959935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885900+0000","last_change":"2024-07-26T06:17:46.877540+0000","last_active":"2024-07-26T06:17:46.885900+0000","last_peered":"2024-07-26T06:17:46.885900+0000","last_clean":"2024-07-26T06:17:46.885900+0000","last_became_active":"2024-07-26T06:17:46.877230+0000","last_became_peered":"2024-07-26T06:17:46.877230+0000","last_unstale":"2024-07-26T06:17:46.885900+0000","last_undegraded":"2024-07-26T06:17:46.885900+0000","last_fullsized":"2024-07-26T06:17:46.885900+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:24:22.437469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.847885+0000","last_change":"2024-07-26T06:17:45.844515+0000","last_active":"2024-07-26T06:17:45.847885+0000","last_peered":"2024-07-26T06:17:45.847885+0000","last_clean":"2024-07-26T06:17:45.847885+0000","last_became_active":"2024-07-26T06:17:45.844194+0000","last_became_peered":"2024-07-26T06:17:45.844194+0000","last_unstale":"2024-07-26T06:17:45.847885+0000","last_undegraded":"2024-07-26T06:17:45.847885+0000","last_fullsized":"2024-07-26T06:17:45.847885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:17.439039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.884158+0000","last_change":"2024-07-26T06:17:46.873820+0000","last_active":"2024-07-26T06:17:46.884158+0000","last_peered":"2024-07-26T06:17:46.884158+0000","last_clean":"2024-07-26T06:17:46.884158+0000","last_became_active":"2024-07-26T06:17:46.873523+0000","last_became_peered":"2024-07-26T06:17:46.873523+0000","last_unstale":"2024-07-26T06:17:46.884158+0000","last_undegraded":"2024-07-26T06:17:46.884158+0000","last_fullsized":"2024-07-26T06:17:46.884158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:51:44.227610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885457+0000","last_change":"2024-07-26T06:17:45.848221+0000","last_active":"2024-07-26T06:17:46.885457+0000","last_peered":"2024-07-26T06:17:46.885457+0000","last_clean":"2024-07-26T06:17:46.885457+0000","last_became_active":"2024-07-26T06:17:45.847745+0000","last_became_peered":"2024-07-26T06:17:45.847745+0000","last_unstale":"2024-07-26T06:17:46.885457+0000","last_undegraded":"2024-07-26T06:17:46.885457+0000","last_fullsized":"2024-07-26T06:17:46.885457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:12:35.713765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.866166+0000","last_change":"2024-07-26T06:17:46.861560+0000","last_active":"2024-07-26T06:17:46.866166+0000","last_peered":"2024-07-26T06:17:46.866166+0000","last_clean":"2024-07-26T06:17:46.866166+0000","last_became_active":"2024-07-26T06:17:46.861277+0000","last_became_peered":"2024-07-26T06:17:46.861277+0000","last_unstale":"2024-07-26T06:17:46.866166+0000","last_undegraded":"2024-07-26T06:17:46.866166+0000","last_fullsized":"2024-07-26T06:17:46.866166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:35.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.839882+0000","last_change":"2024-07-26T06:17:45.833550+0000","last_active":"2024-07-26T06:17:45.839882+0000","last_peered":"2024-07-26T06:17:45.839882+0000","last_clean":"2024-07-26T06:17:45.839882+0000","last_became_active":"2024-07-26T06:17:45.833042+0000","last_became_peered":"2024-07-26T06:17:45.833042+0000","last_unstale":"2024-07-26T06:17:45.839882+0000","last_undegraded":"2024-07-26T06:17:45.839882+0000","last_fullsized":"2024-07-26T06:17:45.839882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:46.976415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.818381+0000","last_change":"2024-07-26T06:17:45.818381+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.818381+0000","last_undegraded":"2024-07-26T06:17:45.818381+0000","last_fullsized":"2024-07-26T06:17:45.818381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886594+0000","last_change":"2024-07-26T06:17:45.847332+0000","last_active":"2024-07-26T06:17:46.886594+0000","last_peered":"2024-07-26T06:17:46.886594+0000","last_clean":"2024-07-26T06:17:46.886594+0000","last_became_active":"2024-07-26T06:17:45.846279+0000","last_became_peered":"2024-07-26T06:17:45.846279+0000","last_unstale":"2024-07-26T06:17:46.886594+0000","last_undegraded":"2024-07-26T06:17:46.886594+0000","last_fullsized":"2024-07-26T06:17:46.886594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:57:35.301365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880496+0000","last_change":"2024-07-26T06:17:46.870578+0000","last_active":"2024-07-26T06:17:46.880496+0000","last_peered":"2024-07-26T06:17:46.880496+0000","last_clean":"2024-07-26T06:17:46.880496+0000","last_became_active":"2024-07-26T06:17:46.870219+0000","last_became_peered":"2024-07-26T06:17:46.870219+0000","last_unstale":"2024-07-26T06:17:46.880496+0000","last_undegraded":"2024-07-26T06:17:46.880496+0000","last_fullsized":"2024-07-26T06:17:46.880496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:27.562507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.843335+0000","last_change":"2024-07-26T06:17:45.839166+0000","last_active":"2024-07-26T06:17:45.843335+0000","last_peered":"2024-07-26T06:17:45.843335+0000","last_clean":"2024-07-26T06:17:45.843335+0000","last_became_active":"2024-07-26T06:17:45.838866+0000","last_became_peered":"2024-07-26T06:17:45.838866+0000","last_unstale":"2024-07-26T06:17:45.843335+0000","last_undegraded":"2024-07-26T06:17:45.843335+0000","last_fullsized":"2024-07-26T06:17:45.843335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:03:51.512511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.871752+0000","last_change":"2024-07-26T06:17:46.869327+0000","last_active":"2024-07-26T06:17:46.871752+0000","last_peered":"2024-07-26T06:17:46.871752+0000","last_clean":"2024-07-26T06:17:46.871752+0000","last_became_active":"2024-07-26T06:17:46.869010+0000","last_became_peered":"2024-07-26T06:17:46.869010+0000","last_unstale":"2024-07-26T06:17:46.871752+0000","last_undegraded":"2024-07-26T06:17:46.871752+0000","last_fullsized":"2024-07-26T06:17:46.871752+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:35.804674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.850350+0000","last_change":"2024-07-26T06:17:45.832174+0000","last_active":"2024-07-26T06:17:46.850350+0000","last_peered":"2024-07-26T06:17:46.850350+0000","last_clean":"2024-07-26T06:17:46.850350+0000","last_became_active":"2024-07-26T06:17:45.831879+0000","last_became_peered":"2024-07-26T06:17:45.831879+0000","last_unstale":"2024-07-26T06:17:46.850350+0000","last_undegraded":"2024-07-26T06:17:46.850350+0000","last_fullsized":"2024-07-26T06:17:46.850350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:37.746116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.874820+0000","last_change":"2024-07-26T06:17:46.872820+0000","last_active":"2024-07-26T06:17:46.874820+0000","last_peered":"2024-07-26T06:17:46.874820+0000","last_clean":"2024-07-26T06:17:46.874820+0000","last_became_active":"2024-07-26T06:17:46.872401+0000","last_became_peered":"2024-07-26T06:17:46.872401+0000","last_unstale":"2024-07-26T06:17:46.874820+0000","last_undegraded":"2024-07-26T06:17:46.874820+0000","last_fullsized":"2024-07-26T06:17:46.874820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:46.672362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.837601+0000","last_change":"2024-07-26T06:17:45.831327+0000","last_active":"2024-07-26T06:17:45.837601+0000","last_peered":"2024-07-26T06:17:45.837601+0000","last_clean":"2024-07-26T06:17:45.837601+0000","last_became_active":"2024-07-26T06:17:45.831042+0000","last_became_peered":"2024-07-26T06:17:45.831042+0000","last_unstale":"2024-07-26T06:17:45.837601+0000","last_undegraded":"2024-07-26T06:17:45.837601+0000","last_fullsized":"2024-07-26T06:17:45.837601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:51:17.440620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887270+0000","last_change":"2024-07-26T06:17:46.877535+0000","last_active":"2024-07-26T06:17:46.887270+0000","last_peered":"2024-07-26T06:17:46.887270+0000","last_clean":"2024-07-26T06:17:46.887270+0000","last_became_active":"2024-07-26T06:17:46.877254+0000","last_became_peered":"2024-07-26T06:17:46.877254+0000","last_unstale":"2024-07-26T06:17:46.887270+0000","last_undegraded":"2024-07-26T06:17:46.887270+0000","last_fullsized":"2024-07-26T06:17:46.887270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:03.446774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870659+0000","last_change":"2024-07-26T06:17:45.849240+0000","last_active":"2024-07-26T06:17:46.870659+0000","last_peered":"2024-07-26T06:17:46.870659+0000","last_clean":"2024-07-26T06:17:46.870659+0000","last_became_active":"2024-07-26T06:17:45.848896+0000","last_became_peered":"2024-07-26T06:17:45.848896+0000","last_unstale":"2024-07-26T06:17:46.870659+0000","last_undegraded":"2024-07-26T06:17:46.870659+0000","last_fullsized":"2024-07-26T06:17:46.870659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:32.867527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885881+0000","last_change":"2024-07-26T06:17:46.876327+0000","last_active":"2024-07-26T06:17:46.885881+0000","last_peered":"2024-07-26T06:17:46.885881+0000","last_clean":"2024-07-26T06:17:46.885881+0000","last_became_active":"2024-07-26T06:17:46.876045+0000","last_became_peered":"2024-07-26T06:17:46.876045+0000","last_unstale":"2024-07-26T06:17:46.885881+0000","last_undegraded":"2024-07-26T06:17:46.885881+0000","last_fullsized":"2024-07-26T06:17:46.885881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:37.615124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.836345+0000","last_change":"2024-07-26T06:17:45.831283+0000","last_active":"2024-07-26T06:17:45.836345+0000","last_peered":"2024-07-26T06:17:45.836345+0000","last_clean":"2024-07-26T06:17:45.836345+0000","last_became_active":"2024-07-26T06:17:45.830822+0000","last_became_peered":"2024-07-26T06:17:45.830822+0000","last_unstale":"2024-07-26T06:17:45.836345+0000","last_undegraded":"2024-07-26T06:17:45.836345+0000","last_fullsized":"2024-07-26T06:17:45.836345+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:34.791150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869815+0000","last_change":"2024-07-26T06:17:46.865363+0000","last_active":"2024-07-26T06:17:46.869815+0000","last_peered":"2024-07-26T06:17:46.869815+0000","last_clean":"2024-07-26T06:17:46.869815+0000","last_became_active":"2024-07-26T06:17:46.865083+0000","last_became_peered":"2024-07-26T06:17:46.865083+0000","last_unstale":"2024-07-26T06:17:46.869815+0000","last_undegraded":"2024-07-26T06:17:46.869815+0000","last_fullsized":"2024-07-26T06:17:46.869815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:00:30.072591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.841012+0000","last_change":"2024-07-26T06:17:45.834795+0000","last_active":"2024-07-26T06:17:45.841012+0000","last_peered":"2024-07-26T06:17:45.841012+0000","last_clean":"2024-07-26T06:17:45.841012+0000","last_became_active":"2024-07-26T06:17:45.834387+0000","last_became_peered":"2024-07-26T06:17:45.834387+0000","last_unstale":"2024-07-26T06:17:45.841012+0000","last_undegraded":"2024-07-26T06:17:45.841012+0000","last_fullsized":"2024-07-26T06:17:45.841012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:03.909872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.875415+0000","last_change":"2024-07-26T06:17:46.867768+0000","last_active":"2024-07-26T06:17:46.875415+0000","last_peered":"2024-07-26T06:17:46.875415+0000","last_clean":"2024-07-26T06:17:46.875415+0000","last_became_active":"2024-07-26T06:17:46.867495+0000","last_became_peered":"2024-07-26T06:17:46.867495+0000","last_unstale":"2024-07-26T06:17:46.875415+0000","last_undegraded":"2024-07-26T06:17:46.875415+0000","last_fullsized":"2024-07-26T06:17:46.875415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:35.804674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.845702+0000","last_change":"2024-07-26T06:17:45.842552+0000","last_active":"2024-07-26T06:17:45.845702+0000","last_peered":"2024-07-26T06:17:45.845702+0000","last_clean":"2024-07-26T06:17:45.845702+0000","last_became_active":"2024-07-26T06:17:45.842252+0000","last_became_peered":"2024-07-26T06:17:45.842252+0000","last_unstale":"2024-07-26T06:17:45.845702+0000","last_undegraded":"2024-07-26T06:17:45.845702+0000","last_fullsized":"2024-07-26T06:17:45.845702+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:49.404319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870085+0000","last_change":"2024-07-26T06:17:46.867179+0000","last_active":"2024-07-26T06:17:46.870085+0000","last_peered":"2024-07-26T06:17:46.870085+0000","last_clean":"2024-07-26T06:17:46.870085+0000","last_became_active":"2024-07-26T06:17:46.866757+0000","last_became_peered":"2024-07-26T06:17:46.866757+0000","last_unstale":"2024-07-26T06:17:46.870085+0000","last_undegraded":"2024-07-26T06:17:46.870085+0000","last_fullsized":"2024-07-26T06:17:46.870085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:06.947898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870468+0000","last_change":"2024-07-26T06:17:45.847778+0000","last_active":"2024-07-26T06:17:46.870468+0000","last_peered":"2024-07-26T06:17:46.870468+0000","last_clean":"2024-07-26T06:17:46.870468+0000","last_became_active":"2024-07-26T06:17:45.846948+0000","last_became_peered":"2024-07-26T06:17:45.846948+0000","last_unstale":"2024-07-26T06:17:46.870468+0000","last_undegraded":"2024-07-26T06:17:46.870468+0000","last_fullsized":"2024-07-26T06:17:46.870468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:34:59.524300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880619+0000","last_change":"2024-07-26T06:17:46.875619+0000","last_active":"2024-07-26T06:17:46.880619+0000","last_peered":"2024-07-26T06:17:46.880619+0000","last_clean":"2024-07-26T06:17:46.880619+0000","last_became_active":"2024-07-26T06:17:46.875342+0000","last_became_peered":"2024-07-26T06:17:46.875342+0000","last_unstale":"2024-07-26T06:17:46.880619+0000","last_undegraded":"2024-07-26T06:17:46.880619+0000","last_fullsized":"2024-07-26T06:17:46.880619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:06.947898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.857240+0000","last_change":"2024-07-26T06:17:45.838946+0000","last_active":"2024-07-26T06:17:46.857240+0000","last_peered":"2024-07-26T06:17:46.857240+0000","last_clean":"2024-07-26T06:17:46.857240+0000","last_became_active":"2024-07-26T06:17:45.837548+0000","last_became_peered":"2024-07-26T06:17:45.837548+0000","last_unstale":"2024-07-26T06:17:46.857240+0000","last_undegraded":"2024-07-26T06:17:46.857240+0000","last_fullsized":"2024-07-26T06:17:46.857240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.120865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.864993+0000","last_change":"2024-07-26T06:17:46.861422+0000","last_active":"2024-07-26T06:17:46.864993+0000","last_peered":"2024-07-26T06:17:46.864993+0000","last_clean":"2024-07-26T06:17:46.864993+0000","last_became_active":"2024-07-26T06:17:46.861110+0000","last_became_peered":"2024-07-26T06:17:46.861110+0000","last_unstale":"2024-07-26T06:17:46.864993+0000","last_undegraded":"2024-07-26T06:17:46.864993+0000","last_fullsized":"2024-07-26T06:17:46.864993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:13:37.507234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.840147+0000","last_change":"2024-07-26T06:17:45.836257+0000","last_active":"2024-07-26T06:17:45.840147+0000","last_peered":"2024-07-26T06:17:45.840147+0000","last_clean":"2024-07-26T06:17:45.840147+0000","last_became_active":"2024-07-26T06:17:45.835965+0000","last_became_peered":"2024-07-26T06:17:45.835965+0000","last_unstale":"2024-07-26T06:17:45.840147+0000","last_undegraded":"2024-07-26T06:17:45.840147+0000","last_fullsized":"2024-07-26T06:17:45.840147+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:24:37.884202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.817267+0000","last_change":"2024-07-26T06:17:45.817267+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.817267+0000","last_undegraded":"2024-07-26T06:17:45.817267+0000","last_fullsized":"2024-07-26T06:17:45.817267+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885398+0000","last_change":"2024-07-26T06:17:45.845767+0000","last_active":"2024-07-26T06:17:46.885398+0000","last_peered":"2024-07-26T06:17:46.885398+0000","last_clean":"2024-07-26T06:17:46.885398+0000","last_became_active":"2024-07-26T06:17:45.845473+0000","last_became_peered":"2024-07-26T06:17:45.845473+0000","last_unstale":"2024-07-26T06:17:46.885398+0000","last_undegraded":"2024-07-26T06:17:46.885398+0000","last_fullsized":"2024-07-26T06:17:46.885398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:35:42.348603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.883735+0000","last_change":"2024-07-26T06:17:46.871870+0000","last_active":"2024-07-26T06:17:46.883735+0000","last_peered":"2024-07-26T06:17:46.883735+0000","last_clean":"2024-07-26T06:17:46.883735+0000","last_became_active":"2024-07-26T06:17:46.871559+0000","last_became_peered":"2024-07-26T06:17:46.871559+0000","last_unstale":"2024-07-26T06:17:46.883735+0000","last_undegraded":"2024-07-26T06:17:46.883735+0000","last_fullsized":"2024-07-26T06:17:46.883735+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:32.260066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.840132+0000","last_change":"2024-07-26T06:17:45.837138+0000","last_active":"2024-07-26T06:17:45.840132+0000","last_peered":"2024-07-26T06:17:45.840132+0000","last_clean":"2024-07-26T06:17:45.840132+0000","last_became_active":"2024-07-26T06:17:45.836830+0000","last_became_peered":"2024-07-26T06:17:45.836830+0000","last_unstale":"2024-07-26T06:17:45.840132+0000","last_undegraded":"2024-07-26T06:17:45.840132+0000","last_fullsized":"2024-07-26T06:17:45.840132+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:52.339095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.825713+0000","last_change":"2024-07-26T06:17:45.825713+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.825713+0000","last_undegraded":"2024-07-26T06:17:45.825713+0000","last_fullsized":"2024-07-26T06:17:45.825713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887686+0000","last_change":"2024-07-26T06:17:45.845107+0000","last_active":"2024-07-26T06:17:46.887686+0000","last_peered":"2024-07-26T06:17:46.887686+0000","last_clean":"2024-07-26T06:17:46.887686+0000","last_became_active":"2024-07-26T06:17:45.844811+0000","last_became_peered":"2024-07-26T06:17:45.844811+0000","last_unstale":"2024-07-26T06:17:46.887686+0000","last_undegraded":"2024-07-26T06:17:46.887686+0000","last_fullsized":"2024-07-26T06:17:46.887686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:15.980042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880437+0000","last_change":"2024-07-26T06:17:46.870832+0000","last_active":"2024-07-26T06:17:46.880437+0000","last_peered":"2024-07-26T06:17:46.880437+0000","last_clean":"2024-07-26T06:17:46.880437+0000","last_became_active":"2024-07-26T06:17:46.870484+0000","last_became_peered":"2024-07-26T06:17:46.870484+0000","last_unstale":"2024-07-26T06:17:46.880437+0000","last_undegraded":"2024-07-26T06:17:46.880437+0000","last_fullsized":"2024-07-26T06:17:46.880437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:17.279152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.860795+0000","last_change":"2024-07-26T06:17:45.842422+0000","last_active":"2024-07-26T06:17:46.860795+0000","last_peered":"2024-07-26T06:17:46.860795+0000","last_clean":"2024-07-26T06:17:46.860795+0000","last_became_active":"2024-07-26T06:17:45.841773+0000","last_became_peered":"2024-07-26T06:17:45.841773+0000","last_unstale":"2024-07-26T06:17:46.860795+0000","last_undegraded":"2024-07-26T06:17:46.860795+0000","last_fullsized":"2024-07-26T06:17:46.860795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:38:26.705105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.811704+0000","last_change":"2024-07-26T06:17:45.811704+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.811704+0000","last_undegraded":"2024-07-26T06:17:45.811704+0000","last_fullsized":"2024-07-26T06:17:45.811704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.875536+0000","last_change":"2024-07-26T06:17:45.847558+0000","last_active":"2024-07-26T06:17:46.875536+0000","last_peered":"2024-07-26T06:17:46.875536+0000","last_clean":"2024-07-26T06:17:46.875536+0000","last_became_active":"2024-07-26T06:17:45.846183+0000","last_became_peered":"2024-07-26T06:17:45.846183+0000","last_unstale":"2024-07-26T06:17:46.875536+0000","last_undegraded":"2024-07-26T06:17:46.875536+0000","last_fullsized":"2024-07-26T06:17:46.875536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:48.622659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.811227+0000","last_change":"2024-07-26T06:17:45.811227+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.811227+0000","last_undegraded":"2024-07-26T06:17:45.811227+0000","last_fullsized":"2024-07-26T06:17:45.811227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.857429+0000","last_change":"2024-07-26T06:17:45.838684+0000","last_active":"2024-07-26T06:17:46.857429+0000","last_peered":"2024-07-26T06:17:46.857429+0000","last_clean":"2024-07-26T06:17:46.857429+0000","last_became_active":"2024-07-26T06:17:45.836740+0000","last_became_peered":"2024-07-26T06:17:45.836740+0000","last_unstale":"2024-07-26T06:17:46.857429+0000","last_undegraded":"2024-07-26T06:17:46.857429+0000","last_fullsized":"2024-07-26T06:17:46.857429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:13.612220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.811242+0000","last_change":"2024-07-26T06:17:45.811242+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.811242+0000","last_undegraded":"2024-07-26T06:17:45.811242+0000","last_fullsized":"2024-07-26T06:17:45.811242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.834886+0000","last_change":"2024-07-26T06:17:45.831813+0000","last_active":"2024-07-26T06:17:46.834886+0000","last_peered":"2024-07-26T06:17:46.834886+0000","last_clean":"2024-07-26T06:17:46.834886+0000","last_became_active":"2024-07-26T06:17:45.831532+0000","last_became_peered":"2024-07-26T06:17:45.831532+0000","last_unstale":"2024-07-26T06:17:46.834886+0000","last_undegraded":"2024-07-26T06:17:46.834886+0000","last_fullsized":"2024-07-26T06:17:46.834886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.120865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.863895+0000","last_change":"2024-07-26T06:17:46.857651+0000","last_active":"2024-07-26T06:17:46.863895+0000","last_peered":"2024-07-26T06:17:46.863895+0000","last_clean":"2024-07-26T06:17:46.863895+0000","last_became_active":"2024-07-26T06:17:46.857133+0000","last_became_peered":"2024-07-26T06:17:46.857133+0000","last_unstale":"2024-07-26T06:17:46.863895+0000","last_undegraded":"2024-07-26T06:17:46.863895+0000","last_fullsized":"2024-07-26T06:17:46.863895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:35.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885309+0000","last_change":"2024-07-26T06:17:45.844087+0000","last_active":"2024-07-26T06:17:46.885309+0000","last_peered":"2024-07-26T06:17:46.885309+0000","last_clean":"2024-07-26T06:17:46.885309+0000","last_became_active":"2024-07-26T06:17:45.843797+0000","last_became_peered":"2024-07-26T06:17:45.843797+0000","last_unstale":"2024-07-26T06:17:46.885309+0000","last_undegraded":"2024-07-26T06:17:46.885309+0000","last_fullsized":"2024-07-26T06:17:46.885309+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:26:26.068716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.866058+0000","last_change":"2024-07-26T06:17:46.860008+0000","last_active":"2024-07-26T06:17:46.866058+0000","last_peered":"2024-07-26T06:17:46.866058+0000","last_clean":"2024-07-26T06:17:46.866058+0000","last_became_active":"2024-07-26T06:17:46.859731+0000","last_became_peered":"2024-07-26T06:17:46.859731+0000","last_unstale":"2024-07-26T06:17:46.866058+0000","last_undegraded":"2024-07-26T06:17:46.866058+0000","last_fullsized":"2024-07-26T06:17:46.866058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:28.166488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.857809+0000","last_change":"2024-07-26T06:17:45.837663+0000","last_active":"2024-07-26T06:17:46.857809+0000","last_peered":"2024-07-26T06:17:46.857809+0000","last_clean":"2024-07-26T06:17:46.857809+0000","last_became_active":"2024-07-26T06:17:45.837276+0000","last_became_peered":"2024-07-26T06:17:45.837276+0000","last_unstale":"2024-07-26T06:17:46.857809+0000","last_undegraded":"2024-07-26T06:17:46.857809+0000","last_fullsized":"2024-07-26T06:17:46.857809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:18.959935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887129+0000","last_change":"2024-07-26T06:17:46.880623+0000","last_active":"2024-07-26T06:17:46.887129+0000","last_peered":"2024-07-26T06:17:46.887129+0000","last_clean":"2024-07-26T06:17:46.887129+0000","last_became_active":"2024-07-26T06:17:46.880329+0000","last_became_peered":"2024-07-26T06:17:46.880329+0000","last_unstale":"2024-07-26T06:17:46.887129+0000","last_undegraded":"2024-07-26T06:17:46.887129+0000","last_fullsized":"2024-07-26T06:17:46.887129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:24.367302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.854853+0000","last_change":"2024-07-26T06:17:45.825394+0000","last_active":"2024-07-26T06:17:46.854853+0000","last_peered":"2024-07-26T06:17:46.854853+0000","last_clean":"2024-07-26T06:17:46.854853+0000","last_became_active":"2024-07-26T06:17:45.824922+0000","last_became_peered":"2024-07-26T06:17:45.824922+0000","last_unstale":"2024-07-26T06:17:46.854853+0000","last_undegraded":"2024-07-26T06:17:46.854853+0000","last_fullsized":"2024-07-26T06:17:46.854853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:01.680799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.883348+0000","last_change":"2024-07-26T06:17:46.871923+0000","last_active":"2024-07-26T06:17:46.883348+0000","last_peered":"2024-07-26T06:17:46.883348+0000","last_clean":"2024-07-26T06:17:46.883348+0000","last_became_active":"2024-07-26T06:17:46.871597+0000","last_became_peered":"2024-07-26T06:17:46.871597+0000","last_unstale":"2024-07-26T06:17:46.883348+0000","last_undegraded":"2024-07-26T06:17:46.883348+0000","last_fullsized":"2024-07-26T06:17:46.883348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:46.672362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.835072+0000","last_change":"2024-07-26T06:17:45.839615+0000","last_active":"2024-07-26T06:17:46.835072+0000","last_peered":"2024-07-26T06:17:46.835072+0000","last_clean":"2024-07-26T06:17:46.835072+0000","last_became_active":"2024-07-26T06:17:45.839286+0000","last_became_peered":"2024-07-26T06:17:45.839286+0000","last_unstale":"2024-07-26T06:17:46.835072+0000","last_undegraded":"2024-07-26T06:17:46.835072+0000","last_fullsized":"2024-07-26T06:17:46.835072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:13.612220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869558+0000","last_change":"2024-07-26T06:17:46.864454+0000","last_active":"2024-07-26T06:17:46.869558+0000","last_peered":"2024-07-26T06:17:46.869558+0000","last_clean":"2024-07-26T06:17:46.869558+0000","last_became_active":"2024-07-26T06:17:46.864188+0000","last_became_peered":"2024-07-26T06:17:46.864188+0000","last_unstale":"2024-07-26T06:17:46.869558+0000","last_undegraded":"2024-07-26T06:17:46.869558+0000","last_fullsized":"2024-07-26T06:17:46.869558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:05:31.790582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.853827+0000","last_change":"2024-07-26T06:17:45.837315+0000","last_active":"2024-07-26T06:17:46.853827+0000","last_peered":"2024-07-26T06:17:46.853827+0000","last_clean":"2024-07-26T06:17:46.853827+0000","last_became_active":"2024-07-26T06:17:45.836993+0000","last_became_peered":"2024-07-26T06:17:45.836993+0000","last_unstale":"2024-07-26T06:17:46.853827+0000","last_undegraded":"2024-07-26T06:17:46.853827+0000","last_fullsized":"2024-07-26T06:17:46.853827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:18.248918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885819+0000","last_change":"2024-07-26T06:17:46.877325+0000","last_active":"2024-07-26T06:17:46.885819+0000","last_peered":"2024-07-26T06:17:46.885819+0000","last_clean":"2024-07-26T06:17:46.885819+0000","last_became_active":"2024-07-26T06:17:46.877004+0000","last_became_peered":"2024-07-26T06:17:46.877004+0000","last_unstale":"2024-07-26T06:17:46.885819+0000","last_undegraded":"2024-07-26T06:17:46.885819+0000","last_fullsized":"2024-07-26T06:17:46.885819+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:58.687274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.843314+0000","last_change":"2024-07-26T06:17:45.838377+0000","last_active":"2024-07-26T06:17:45.843314+0000","last_peered":"2024-07-26T06:17:45.843314+0000","last_clean":"2024-07-26T06:17:45.843314+0000","last_became_active":"2024-07-26T06:17:45.838098+0000","last_became_peered":"2024-07-26T06:17:45.838098+0000","last_unstale":"2024-07-26T06:17:45.843314+0000","last_undegraded":"2024-07-26T06:17:45.843314+0000","last_fullsized":"2024-07-26T06:17:45.843314+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:56:19.677606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.825355+0000","last_change":"2024-07-26T06:17:45.825355+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.825355+0000","last_undegraded":"2024-07-26T06:17:45.825355+0000","last_fullsized":"2024-07-26T06:17:45.825355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-07-26T06:17:45.843535+0000","last_change":"2024-07-26T06:17:45.840975+0000","last_active":"2024-07-26T06:17:45.843535+0000","last_peered":"2024-07-26T06:17:45.843535+0000","last_clean":"2024-07-26T06:17:45.843535+0000","last_became_active":"2024-07-26T06:17:45.840717+0000","last_became_peered":"2024-07-26T06:17:45.840717+0000","last_unstale":"2024-07-26T06:17:45.843535+0000","last_undegraded":"2024-07-26T06:17:45.843535+0000","last_fullsized":"2024-07-26T06:17:45.843535+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:54:09.083387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870138+0000","last_change":"2024-07-26T06:17:46.862600+0000","last_active":"2024-07-26T06:17:46.870138+0000","last_peered":"2024-07-26T06:17:46.870138+0000","last_clean":"2024-07-26T06:17:46.870138+0000","last_became_active":"2024-07-26T06:17:46.862055+0000","last_became_peered":"2024-07-26T06:17:46.862055+0000","last_unstale":"2024-07-26T06:17:46.870138+0000","last_undegraded":"2024-07-26T06:17:46.870138+0000","last_fullsized":"2024-07-26T06:17:46.870138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:54:38.354576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.871666+0000","last_change":"2024-07-26T06:17:45.840005+0000","last_active":"2024-07-26T06:17:46.871666+0000","last_peered":"2024-07-26T06:17:46.871666+0000","last_clean":"2024-07-26T06:17:46.871666+0000","last_became_active":"2024-07-26T06:17:45.839706+0000","last_became_peered":"2024-07-26T06:17:45.839706+0000","last_unstale":"2024-07-26T06:17:46.871666+0000","last_undegraded":"2024-07-26T06:17:46.871666+0000","last_fullsized":"2024-07-26T06:17:46.871666+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:22:45.516586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-07-26T06:17:45.811294+0000","last_change":"2024-07-26T06:17:45.811294+0000","last_active":"2024-07-26T06:17:45.798295+0000","last_peered":"2024-07-26T06:17:45.798295+0000","last_clean":"2024-07-26T06:17:45.798295+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:17:45.811294+0000","last_undegraded":"2024-07-26T06:17:45.811294+0000","last_fullsized":"2024-07-26T06:17:45.811294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885644+0000","last_change":"2024-07-26T06:17:45.847226+0000","last_active":"2024-07-26T06:17:46.885644+0000","last_peered":"2024-07-26T06:17:46.885644+0000","last_clean":"2024-07-26T06:17:46.885644+0000","last_became_active":"2024-07-26T06:17:45.846508+0000","last_became_peered":"2024-07-26T06:17:45.846508+0000","last_unstale":"2024-07-26T06:17:46.885644+0000","last_undegraded":"2024-07-26T06:17:46.885644+0000","last_fullsized":"2024-07-26T06:17:46.885644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:32:10.105503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":196,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.853866+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:17:46.853866+0000","last_peered":"2024-07-26T06:17:46.853866+0000","last_clean":"2024-07-26T06:17:46.853866+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:17:46.853866+0000","last_undegraded":"2024-07-26T06:17:46.853866+0000","last_fullsized":"2024-07-26T06:17:46.853866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":674,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":933939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":336180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509496,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":823296,"data_stored":343045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":344567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1413120,"data_stored":933323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":336180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":329905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":329905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:17:51.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:17:51.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 -- 172.21.15.163:0/1707566827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350015b390 msgr2=0x7f350015b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 --2- 172.21.15.163:0/1707566827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350015b390 0x7f350015b830 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f34e8005fd0 tx=0x7f34e802a540 comp rx=0 tx=0).stop 2024-07-26T06:17:51.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 -- 172.21.15.163:0/1707566827 shutdown_connections 2024-07-26T06:17:51.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 --2- 172.21.15.163:0/1707566827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f350006bf40 0x7f350006c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 --2- 172.21.15.163:0/1707566827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350015b390 0x7f350015b830 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 -- 172.21.15.163:0/1707566827 >> 172.21.15.163:0/1707566827 conn(0x7f350006d420 msgr2=0x7f350006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:51.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 -- 172.21.15.163:0/1707566827 shutdown_connections 2024-07-26T06:17:51.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.536+0000 7f3506c00640 1 -- 172.21.15.163:0/1707566827 wait complete. 2024-07-26T06:17:51.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.539+0000 7f3506c00640 1 Processor -- start 2024-07-26T06:17:51.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.539+0000 7f3506c00640 1 -- start start 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f3506c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 0x7f3500149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f3506c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35001545e0 0x7f3500143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f3506c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f350015b390 0x7f35000b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f3506c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3500072f20 con 0x7f350006bf40 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f3506c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3500072da0 con 0x7f350015b390 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f3506c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35000730a0 con 0x7f35001545e0 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f34ffe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 0x7f3500149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f34ffe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 0x7f3500149e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34200/0 (socket says 172.21.15.163:34200) 2024-07-26T06:17:51.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.540+0000 7f34ffe00640 1 -- 172.21.15.163:0/4238434616 learned_addr learned my addr 172.21.15.163:0/4238434616 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:51.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34ffe00640 1 -- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35001545e0 msgr2=0x7f3500143be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34ffe00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35001545e0 0x7f3500143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34ffe00640 1 -- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f350015b390 msgr2=0x7f35000b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34ffe00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f350015b390 0x7f35000b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34ffe00640 1 -- 172.21.15.163:0/4238434616 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35001465c0 con 0x7f350006bf40 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34ffe00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 0x7f3500149e60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f34f000c430 tx=0x7f34f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0011790 con 0x7f350006bf40 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34f0017070 con 0x7f350006bf40 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35001468a0 con 0x7f350006bf40 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0011d00 con 0x7f350006bf40 2024-07-26T06:17:51.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.541+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3500146e60 con 0x7f350006bf40 2024-07-26T06:17:51.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.542+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350006c370 con 0x7f350006bf40 2024-07-26T06:17:51.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.544+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f34f001c460 con 0x7f350006bf40 2024-07-26T06:17:51.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.545+0000 7f34f7e00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f34dc076630 0x7f34dc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:51.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.545+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f34f0097bc0 con 0x7f350006bf40 2024-07-26T06:17:51.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.545+0000 7f34ff400640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f34dc076630 0x7f34dc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:51.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.546+0000 7f34ff400640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f34dc076630 0x7f34dc078af0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34e8033000 tx=0x7f34e80046c0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:51.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.549+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f34f00611a0 con 0x7f350006bf40 2024-07-26T06:17:51.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.751+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f35000b11d0 con 0x7f34dc076630 2024-07-26T06:17:51.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.763+0000 7f34f7e00640 1 -- 172.21.15.163:0/4238434616 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252012 (secure 0 0 0) 0x7f35000b11d0 con 0x7f34dc076630 2024-07-26T06:17:51.767 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:51.769 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:17:51.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f34dc076630 msgr2=0x7f34dc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f34dc076630 0x7f34dc078af0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34e8033000 tx=0x7f34e80046c0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 msgr2=0x7f3500149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 0x7f3500149e60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f34f000c430 tx=0x7f34f000c8f0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 shutdown_connections 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f34dc076630 0x7f34dc078af0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f350015b390 0x7f35000b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35001545e0 0x7f3500143be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 --2- 172.21.15.163:0/4238434616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f350006bf40 0x7f3500149e60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 >> 172.21.15.163:0/4238434616 conn(0x7f350006d420 msgr2=0x7f3500073e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 shutdown_connections 2024-07-26T06:17:51.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:51.769+0000 7f3506c00640 1 -- 172.21.15.163:0/4238434616 wait complete. 2024-07-26T06:17:51.790 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-07-26T06:17:51.742740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252944,"kb_used_data":7640,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721776,"statfs":{"total":773094113280,"available":772835098624,"internally_reserved":0,"allocated":7823360,"data_stored":3941229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001974"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.877704+0000","last_change":"2024-07-26T06:17:46.868114+0000","last_active":"2024-07-26T06:17:46.877704+0000","last_peered":"2024-07-26T06:17:46.877704+0000","last_clean":"2024-07-26T06:17:46.877704+0000","last_became_active":"2024-07-26T06:17:46.867785+0000","last_became_peered":"2024-07-26T06:17:46.867785+0000","last_unstale":"2024-07-26T06:17:46.877704+0000","last_undegraded":"2024-07-26T06:17:46.877704+0000","last_fullsized":"2024-07-26T06:17:46.877704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:06.947898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.878455+0000","last_change":"2024-07-26T06:17:46.872554+0000","last_active":"2024-07-26T06:17:46.878455+0000","last_peered":"2024-07-26T06:17:46.878455+0000","last_clean":"2024-07-26T06:17:46.878455+0000","last_became_active":"2024-07-26T06:17:46.871064+0000","last_became_peered":"2024-07-26T06:17:46.871064+0000","last_unstale":"2024-07-26T06:17:46.878455+0000","last_undegraded":"2024-07-26T06:17:46.878455+0000","last_fullsized":"2024-07-26T06:17:46.878455+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:28.285738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.582181+0000","last_change":"2024-07-26T06:17:46.852518+0000","last_active":"2024-07-26T06:17:49.582181+0000","last_peered":"2024-07-26T06:17:49.582181+0000","last_clean":"2024-07-26T06:17:49.582181+0000","last_became_active":"2024-07-26T06:17:46.852227+0000","last_became_peered":"2024-07-26T06:17:46.852227+0000","last_unstale":"2024-07-26T06:17:49.582181+0000","last_undegraded":"2024-07-26T06:17:49.582181+0000","last_fullsized":"2024-07-26T06:17:49.582181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:53:14.307451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640474+0000","last_change":"2024-07-26T06:17:46.878595+0000","last_active":"2024-07-26T06:17:48.640474+0000","last_peered":"2024-07-26T06:17:48.640474+0000","last_clean":"2024-07-26T06:17:48.640474+0000","last_became_active":"2024-07-26T06:17:46.878042+0000","last_became_peered":"2024-07-26T06:17:46.878042+0000","last_unstale":"2024-07-26T06:17:48.640474+0000","last_undegraded":"2024-07-26T06:17:48.640474+0000","last_fullsized":"2024-07-26T06:17:48.640474+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:46.825976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880634+0000","last_change":"2024-07-26T06:17:46.872713+0000","last_active":"2024-07-26T06:17:46.880634+0000","last_peered":"2024-07-26T06:17:46.880634+0000","last_clean":"2024-07-26T06:17:46.880634+0000","last_became_active":"2024-07-26T06:17:46.871978+0000","last_became_peered":"2024-07-26T06:17:46.871978+0000","last_unstale":"2024-07-26T06:17:46.880634+0000","last_undegraded":"2024-07-26T06:17:46.880634+0000","last_fullsized":"2024-07-26T06:17:46.880634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:30.620660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886672+0000","last_change":"2024-07-26T06:17:46.880828+0000","last_active":"2024-07-26T06:17:46.886672+0000","last_peered":"2024-07-26T06:17:46.886672+0000","last_clean":"2024-07-26T06:17:46.886672+0000","last_became_active":"2024-07-26T06:17:46.875622+0000","last_became_peered":"2024-07-26T06:17:46.875622+0000","last_unstale":"2024-07-26T06:17:46.886672+0000","last_undegraded":"2024-07-26T06:17:46.886672+0000","last_fullsized":"2024-07-26T06:17:46.886672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:08.561347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869796+0000","last_change":"2024-07-26T06:17:46.865079+0000","last_active":"2024-07-26T06:17:46.869796+0000","last_peered":"2024-07-26T06:17:46.869796+0000","last_clean":"2024-07-26T06:17:46.869796+0000","last_became_active":"2024-07-26T06:17:46.864008+0000","last_became_peered":"2024-07-26T06:17:46.864008+0000","last_unstale":"2024-07-26T06:17:46.869796+0000","last_undegraded":"2024-07-26T06:17:46.869796+0000","last_fullsized":"2024-07-26T06:17:46.869796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:35.804674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.881946+0000","last_change":"2024-07-26T06:17:46.871894+0000","last_active":"2024-07-26T06:17:46.881946+0000","last_peered":"2024-07-26T06:17:46.881946+0000","last_clean":"2024-07-26T06:17:46.881946+0000","last_became_active":"2024-07-26T06:17:46.870785+0000","last_became_peered":"2024-07-26T06:17:46.870785+0000","last_unstale":"2024-07-26T06:17:46.881946+0000","last_undegraded":"2024-07-26T06:17:46.881946+0000","last_fullsized":"2024-07-26T06:17:46.881946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:04:18.146763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880646+0000","last_change":"2024-07-26T06:17:46.872337+0000","last_active":"2024-07-26T06:17:46.880646+0000","last_peered":"2024-07-26T06:17:46.880646+0000","last_clean":"2024-07-26T06:17:46.880646+0000","last_became_active":"2024-07-26T06:17:46.872037+0000","last_became_peered":"2024-07-26T06:17:46.872037+0000","last_unstale":"2024-07-26T06:17:46.880646+0000","last_undegraded":"2024-07-26T06:17:46.880646+0000","last_fullsized":"2024-07-26T06:17:46.880646+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:18:52.079504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142596+0000","last_change":"2024-07-26T06:17:46.862920+0000","last_active":"2024-07-26T06:17:48.142596+0000","last_peered":"2024-07-26T06:17:48.142596+0000","last_clean":"2024-07-26T06:17:48.142596+0000","last_became_active":"2024-07-26T06:17:46.860634+0000","last_became_peered":"2024-07-26T06:17:46.860634+0000","last_unstale":"2024-07-26T06:17:48.142596+0000","last_undegraded":"2024-07-26T06:17:48.142596+0000","last_fullsized":"2024-07-26T06:17:48.142596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:28.843865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886860+0000","last_change":"2024-07-26T06:17:46.874454+0000","last_active":"2024-07-26T06:17:46.886860+0000","last_peered":"2024-07-26T06:17:46.886860+0000","last_clean":"2024-07-26T06:17:46.886860+0000","last_became_active":"2024-07-26T06:17:46.874142+0000","last_became_peered":"2024-07-26T06:17:46.874142+0000","last_unstale":"2024-07-26T06:17:46.886860+0000","last_undegraded":"2024-07-26T06:17:46.886860+0000","last_fullsized":"2024-07-26T06:17:46.886860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:37:09.636337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640447+0000","last_change":"2024-07-26T06:17:46.858806+0000","last_active":"2024-07-26T06:17:48.640447+0000","last_peered":"2024-07-26T06:17:48.640447+0000","last_clean":"2024-07-26T06:17:48.640447+0000","last_became_active":"2024-07-26T06:17:46.858197+0000","last_became_peered":"2024-07-26T06:17:46.858197+0000","last_unstale":"2024-07-26T06:17:48.640447+0000","last_undegraded":"2024-07-26T06:17:48.640447+0000","last_fullsized":"2024-07-26T06:17:48.640447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:44:31.769157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880811+0000","last_change":"2024-07-26T06:17:46.877319+0000","last_active":"2024-07-26T06:17:46.880811+0000","last_peered":"2024-07-26T06:17:46.880811+0000","last_clean":"2024-07-26T06:17:46.880811+0000","last_became_active":"2024-07-26T06:17:46.877008+0000","last_became_peered":"2024-07-26T06:17:46.877008+0000","last_unstale":"2024-07-26T06:17:46.880811+0000","last_undegraded":"2024-07-26T06:17:46.880811+0000","last_fullsized":"2024-07-26T06:17:46.880811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:46.672362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869852+0000","last_change":"2024-07-26T06:17:46.865837+0000","last_active":"2024-07-26T06:17:46.869852+0000","last_peered":"2024-07-26T06:17:46.869852+0000","last_clean":"2024-07-26T06:17:46.869852+0000","last_became_active":"2024-07-26T06:17:46.865510+0000","last_became_peered":"2024-07-26T06:17:46.865510+0000","last_unstale":"2024-07-26T06:17:46.869852+0000","last_undegraded":"2024-07-26T06:17:46.869852+0000","last_fullsized":"2024-07-26T06:17:46.869852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:36:54.501776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.863933+0000","last_change":"2024-07-26T06:17:46.857332+0000","last_active":"2024-07-26T06:17:46.863933+0000","last_peered":"2024-07-26T06:17:46.863933+0000","last_clean":"2024-07-26T06:17:46.863933+0000","last_became_active":"2024-07-26T06:17:46.857021+0000","last_became_peered":"2024-07-26T06:17:46.857021+0000","last_unstale":"2024-07-26T06:17:46.863933+0000","last_undegraded":"2024-07-26T06:17:46.863933+0000","last_fullsized":"2024-07-26T06:17:46.863933+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:28.166488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.141994+0000","last_change":"2024-07-26T06:17:46.878589+0000","last_active":"2024-07-26T06:17:48.141994+0000","last_peered":"2024-07-26T06:17:48.141994+0000","last_clean":"2024-07-26T06:17:48.141994+0000","last_became_active":"2024-07-26T06:17:46.878216+0000","last_became_peered":"2024-07-26T06:17:46.878216+0000","last_unstale":"2024-07-26T06:17:48.141994+0000","last_undegraded":"2024-07-26T06:17:48.141994+0000","last_fullsized":"2024-07-26T06:17:48.141994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:53.959594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880665+0000","last_change":"2024-07-26T06:17:46.872692+0000","last_active":"2024-07-26T06:17:46.880665+0000","last_peered":"2024-07-26T06:17:46.880665+0000","last_clean":"2024-07-26T06:17:46.880665+0000","last_became_active":"2024-07-26T06:17:46.872397+0000","last_became_peered":"2024-07-26T06:17:46.872397+0000","last_unstale":"2024-07-26T06:17:46.880665+0000","last_undegraded":"2024-07-26T06:17:46.880665+0000","last_fullsized":"2024-07-26T06:17:46.880665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:54.857970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640963+0000","last_change":"2024-07-26T06:17:46.874405+0000","last_active":"2024-07-26T06:17:48.640963+0000","last_peered":"2024-07-26T06:17:48.640963+0000","last_clean":"2024-07-26T06:17:48.640963+0000","last_became_active":"2024-07-26T06:17:46.873802+0000","last_became_peered":"2024-07-26T06:17:46.873802+0000","last_unstale":"2024-07-26T06:17:48.640963+0000","last_undegraded":"2024-07-26T06:17:48.640963+0000","last_fullsized":"2024-07-26T06:17:48.640963+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:27:52.393729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886904+0000","last_change":"2024-07-26T06:17:46.876973+0000","last_active":"2024-07-26T06:17:46.886904+0000","last_peered":"2024-07-26T06:17:46.886904+0000","last_clean":"2024-07-26T06:17:46.886904+0000","last_became_active":"2024-07-26T06:17:46.876656+0000","last_became_peered":"2024-07-26T06:17:46.876656+0000","last_unstale":"2024-07-26T06:17:46.886904+0000","last_undegraded":"2024-07-26T06:17:46.886904+0000","last_fullsized":"2024-07-26T06:17:46.886904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:34.566927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.871423+0000","last_change":"2024-07-26T06:17:46.864998+0000","last_active":"2024-07-26T06:17:46.871423+0000","last_peered":"2024-07-26T06:17:46.871423+0000","last_clean":"2024-07-26T06:17:46.871423+0000","last_became_active":"2024-07-26T06:17:46.864705+0000","last_became_peered":"2024-07-26T06:17:46.864705+0000","last_unstale":"2024-07-26T06:17:46.871423+0000","last_undegraded":"2024-07-26T06:17:46.871423+0000","last_fullsized":"2024-07-26T06:17:46.871423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:38:27.710065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142389+0000","last_change":"2024-07-26T06:17:46.851206+0000","last_active":"2024-07-26T06:17:48.142389+0000","last_peered":"2024-07-26T06:17:48.142389+0000","last_clean":"2024-07-26T06:17:48.142389+0000","last_became_active":"2024-07-26T06:17:46.850447+0000","last_became_peered":"2024-07-26T06:17:46.850447+0000","last_unstale":"2024-07-26T06:17:48.142389+0000","last_undegraded":"2024-07-26T06:17:48.142389+0000","last_fullsized":"2024-07-26T06:17:48.142389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:18.108652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142122+0000","last_change":"2024-07-26T06:17:46.864952+0000","last_active":"2024-07-26T06:17:48.142122+0000","last_peered":"2024-07-26T06:17:48.142122+0000","last_clean":"2024-07-26T06:17:48.142122+0000","last_became_active":"2024-07-26T06:17:46.864604+0000","last_became_peered":"2024-07-26T06:17:46.864604+0000","last_unstale":"2024-07-26T06:17:48.142122+0000","last_undegraded":"2024-07-26T06:17:48.142122+0000","last_fullsized":"2024-07-26T06:17:48.142122+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:42:00.528565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.883610+0000","last_change":"2024-07-26T06:17:46.872123+0000","last_active":"2024-07-26T06:17:46.883610+0000","last_peered":"2024-07-26T06:17:46.883610+0000","last_clean":"2024-07-26T06:17:46.883610+0000","last_became_active":"2024-07-26T06:17:46.871778+0000","last_became_peered":"2024-07-26T06:17:46.871778+0000","last_unstale":"2024-07-26T06:17:46.883610+0000","last_undegraded":"2024-07-26T06:17:46.883610+0000","last_fullsized":"2024-07-26T06:17:46.883610+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:28.360976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887063+0000","last_change":"2024-07-26T06:17:46.879628+0000","last_active":"2024-07-26T06:17:46.887063+0000","last_peered":"2024-07-26T06:17:46.887063+0000","last_clean":"2024-07-26T06:17:46.887063+0000","last_became_active":"2024-07-26T06:17:46.879345+0000","last_became_peered":"2024-07-26T06:17:46.879345+0000","last_unstale":"2024-07-26T06:17:46.887063+0000","last_undegraded":"2024-07-26T06:17:46.887063+0000","last_fullsized":"2024-07-26T06:17:46.887063+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:31:34.735640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142945+0000","last_change":"2024-07-26T06:17:46.869402+0000","last_active":"2024-07-26T06:17:48.142945+0000","last_peered":"2024-07-26T06:17:48.142945+0000","last_clean":"2024-07-26T06:17:48.142945+0000","last_became_active":"2024-07-26T06:17:46.869123+0000","last_became_peered":"2024-07-26T06:17:46.869123+0000","last_unstale":"2024-07-26T06:17:48.142945+0000","last_undegraded":"2024-07-26T06:17:48.142945+0000","last_fullsized":"2024-07-26T06:17:48.142945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:35:48.631333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886724+0000","last_change":"2024-07-26T06:17:46.884327+0000","last_active":"2024-07-26T06:17:46.886724+0000","last_peered":"2024-07-26T06:17:46.886724+0000","last_clean":"2024-07-26T06:17:46.886724+0000","last_became_active":"2024-07-26T06:17:46.882222+0000","last_became_peered":"2024-07-26T06:17:46.882222+0000","last_unstale":"2024-07-26T06:17:46.886724+0000","last_undegraded":"2024-07-26T06:17:46.886724+0000","last_fullsized":"2024-07-26T06:17:46.886724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:32.260066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.582315+0000","last_change":"2024-07-26T06:17:46.868029+0000","last_active":"2024-07-26T06:17:49.582315+0000","last_peered":"2024-07-26T06:17:49.582315+0000","last_clean":"2024-07-26T06:17:49.582315+0000","last_became_active":"2024-07-26T06:17:46.867637+0000","last_became_peered":"2024-07-26T06:17:46.867637+0000","last_unstale":"2024-07-26T06:17:49.582315+0000","last_undegraded":"2024-07-26T06:17:49.582315+0000","last_fullsized":"2024-07-26T06:17:49.582315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:01.809379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142943+0000","last_change":"2024-07-26T06:17:46.863998+0000","last_active":"2024-07-26T06:17:48.142943+0000","last_peered":"2024-07-26T06:17:48.142943+0000","last_clean":"2024-07-26T06:17:48.142943+0000","last_became_active":"2024-07-26T06:17:46.863477+0000","last_became_peered":"2024-07-26T06:17:46.863477+0000","last_unstale":"2024-07-26T06:17:48.142943+0000","last_undegraded":"2024-07-26T06:17:48.142943+0000","last_fullsized":"2024-07-26T06:17:48.142943+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:40.208607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.581759+0000","last_change":"2024-07-26T06:17:46.863671+0000","last_active":"2024-07-26T06:17:49.581759+0000","last_peered":"2024-07-26T06:17:49.581759+0000","last_clean":"2024-07-26T06:17:49.581759+0000","last_became_active":"2024-07-26T06:17:46.863115+0000","last_became_peered":"2024-07-26T06:17:46.863115+0000","last_unstale":"2024-07-26T06:17:49.581759+0000","last_undegraded":"2024-07-26T06:17:49.581759+0000","last_fullsized":"2024-07-26T06:17:49.581759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:09:49.966656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886875+0000","last_change":"2024-07-26T06:17:46.883094+0000","last_active":"2024-07-26T06:17:46.886875+0000","last_peered":"2024-07-26T06:17:46.886875+0000","last_clean":"2024-07-26T06:17:46.886875+0000","last_became_active":"2024-07-26T06:17:46.882798+0000","last_became_peered":"2024-07-26T06:17:46.882798+0000","last_unstale":"2024-07-26T06:17:46.886875+0000","last_undegraded":"2024-07-26T06:17:46.886875+0000","last_fullsized":"2024-07-26T06:17:46.886875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:28:28.360976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869019+0000","last_change":"2024-07-26T06:17:46.864323+0000","last_active":"2024-07-26T06:17:46.869019+0000","last_peered":"2024-07-26T06:17:46.869019+0000","last_clean":"2024-07-26T06:17:46.869019+0000","last_became_active":"2024-07-26T06:17:46.863716+0000","last_became_peered":"2024-07-26T06:17:46.863716+0000","last_unstale":"2024-07-26T06:17:46.869019+0000","last_undegraded":"2024-07-26T06:17:46.869019+0000","last_fullsized":"2024-07-26T06:17:46.869019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:38:27.710065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.881926+0000","last_change":"2024-07-26T06:17:46.871077+0000","last_active":"2024-07-26T06:17:46.881926+0000","last_peered":"2024-07-26T06:17:46.881926+0000","last_clean":"2024-07-26T06:17:46.881926+0000","last_became_active":"2024-07-26T06:17:46.869322+0000","last_became_peered":"2024-07-26T06:17:46.869322+0000","last_unstale":"2024-07-26T06:17:46.881926+0000","last_undegraded":"2024-07-26T06:17:46.881926+0000","last_fullsized":"2024-07-26T06:17:46.881926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:28.015264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887161+0000","last_change":"2024-07-26T06:17:46.881105+0000","last_active":"2024-07-26T06:17:46.887161+0000","last_peered":"2024-07-26T06:17:46.887161+0000","last_clean":"2024-07-26T06:17:46.887161+0000","last_became_active":"2024-07-26T06:17:46.880809+0000","last_became_peered":"2024-07-26T06:17:46.880809+0000","last_unstale":"2024-07-26T06:17:46.887161+0000","last_undegraded":"2024-07-26T06:17:46.887161+0000","last_fullsized":"2024-07-26T06:17:46.887161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:41:51.520946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886637+0000","last_change":"2024-07-26T06:17:45.848547+0000","last_active":"2024-07-26T06:17:46.886637+0000","last_peered":"2024-07-26T06:17:46.886637+0000","last_clean":"2024-07-26T06:17:46.886637+0000","last_became_active":"2024-07-26T06:17:45.848169+0000","last_became_peered":"2024-07-26T06:17:45.848169+0000","last_unstale":"2024-07-26T06:17:46.886637+0000","last_undegraded":"2024-07-26T06:17:46.886637+0000","last_fullsized":"2024-07-26T06:17:46.886637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:34.260020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880622+0000","last_change":"2024-07-26T06:17:46.871455+0000","last_active":"2024-07-26T06:17:46.880622+0000","last_peered":"2024-07-26T06:17:46.880622+0000","last_clean":"2024-07-26T06:17:46.880622+0000","last_became_active":"2024-07-26T06:17:46.871207+0000","last_became_peered":"2024-07-26T06:17:46.871207+0000","last_unstale":"2024-07-26T06:17:46.880622+0000","last_undegraded":"2024-07-26T06:17:46.880622+0000","last_fullsized":"2024-07-26T06:17:46.880622+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:46.694674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880977+0000","last_change":"2024-07-26T06:17:45.831881+0000","last_active":"2024-07-26T06:17:46.880977+0000","last_peered":"2024-07-26T06:17:46.880977+0000","last_clean":"2024-07-26T06:17:46.880977+0000","last_became_active":"2024-07-26T06:17:45.831631+0000","last_became_peered":"2024-07-26T06:17:45.831631+0000","last_unstale":"2024-07-26T06:17:46.880977+0000","last_undegraded":"2024-07-26T06:17:46.880977+0000","last_fullsized":"2024-07-26T06:17:46.880977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:18.959935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885900+0000","last_change":"2024-07-26T06:17:46.877540+0000","last_active":"2024-07-26T06:17:46.885900+0000","last_peered":"2024-07-26T06:17:46.885900+0000","last_clean":"2024-07-26T06:17:46.885900+0000","last_became_active":"2024-07-26T06:17:46.877230+0000","last_became_peered":"2024-07-26T06:17:46.877230+0000","last_unstale":"2024-07-26T06:17:46.885900+0000","last_undegraded":"2024-07-26T06:17:46.885900+0000","last_fullsized":"2024-07-26T06:17:46.885900+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:24:22.437469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142382+0000","last_change":"2024-07-26T06:17:45.844515+0000","last_active":"2024-07-26T06:17:48.142382+0000","last_peered":"2024-07-26T06:17:48.142382+0000","last_clean":"2024-07-26T06:17:48.142382+0000","last_became_active":"2024-07-26T06:17:45.844194+0000","last_became_peered":"2024-07-26T06:17:45.844194+0000","last_unstale":"2024-07-26T06:17:48.142382+0000","last_undegraded":"2024-07-26T06:17:48.142382+0000","last_fullsized":"2024-07-26T06:17:48.142382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:27:51.388769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.884158+0000","last_change":"2024-07-26T06:17:46.873820+0000","last_active":"2024-07-26T06:17:46.884158+0000","last_peered":"2024-07-26T06:17:46.884158+0000","last_clean":"2024-07-26T06:17:46.884158+0000","last_became_active":"2024-07-26T06:17:46.873523+0000","last_became_peered":"2024-07-26T06:17:46.873523+0000","last_unstale":"2024-07-26T06:17:46.884158+0000","last_undegraded":"2024-07-26T06:17:46.884158+0000","last_fullsized":"2024-07-26T06:17:46.884158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:51:44.227610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885457+0000","last_change":"2024-07-26T06:17:45.848221+0000","last_active":"2024-07-26T06:17:46.885457+0000","last_peered":"2024-07-26T06:17:46.885457+0000","last_clean":"2024-07-26T06:17:46.885457+0000","last_became_active":"2024-07-26T06:17:45.847745+0000","last_became_peered":"2024-07-26T06:17:45.847745+0000","last_unstale":"2024-07-26T06:17:46.885457+0000","last_undegraded":"2024-07-26T06:17:46.885457+0000","last_fullsized":"2024-07-26T06:17:46.885457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:12:35.713765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.866166+0000","last_change":"2024-07-26T06:17:46.861560+0000","last_active":"2024-07-26T06:17:46.866166+0000","last_peered":"2024-07-26T06:17:46.866166+0000","last_clean":"2024-07-26T06:17:46.866166+0000","last_became_active":"2024-07-26T06:17:46.861277+0000","last_became_peered":"2024-07-26T06:17:46.861277+0000","last_unstale":"2024-07-26T06:17:46.866166+0000","last_undegraded":"2024-07-26T06:17:46.866166+0000","last_fullsized":"2024-07-26T06:17:46.866166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:35.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.582095+0000","last_change":"2024-07-26T06:17:45.833550+0000","last_active":"2024-07-26T06:17:49.582095+0000","last_peered":"2024-07-26T06:17:49.582095+0000","last_clean":"2024-07-26T06:17:49.582095+0000","last_became_active":"2024-07-26T06:17:45.833042+0000","last_became_peered":"2024-07-26T06:17:45.833042+0000","last_unstale":"2024-07-26T06:17:49.582095+0000","last_undegraded":"2024-07-26T06:17:49.582095+0000","last_fullsized":"2024-07-26T06:17:49.582095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:06:25.368928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142911+0000","last_change":"2024-07-26T06:17:46.874419+0000","last_active":"2024-07-26T06:17:48.142911+0000","last_peered":"2024-07-26T06:17:48.142911+0000","last_clean":"2024-07-26T06:17:48.142911+0000","last_became_active":"2024-07-26T06:17:46.874111+0000","last_became_peered":"2024-07-26T06:17:46.874111+0000","last_unstale":"2024-07-26T06:17:48.142911+0000","last_undegraded":"2024-07-26T06:17:48.142911+0000","last_fullsized":"2024-07-26T06:17:48.142911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:22.719636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.886594+0000","last_change":"2024-07-26T06:17:45.847332+0000","last_active":"2024-07-26T06:17:46.886594+0000","last_peered":"2024-07-26T06:17:46.886594+0000","last_clean":"2024-07-26T06:17:46.886594+0000","last_became_active":"2024-07-26T06:17:45.846279+0000","last_became_peered":"2024-07-26T06:17:45.846279+0000","last_unstale":"2024-07-26T06:17:46.886594+0000","last_undegraded":"2024-07-26T06:17:46.886594+0000","last_fullsized":"2024-07-26T06:17:46.886594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:57:35.301365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880496+0000","last_change":"2024-07-26T06:17:46.870578+0000","last_active":"2024-07-26T06:17:46.880496+0000","last_peered":"2024-07-26T06:17:46.880496+0000","last_clean":"2024-07-26T06:17:46.880496+0000","last_became_active":"2024-07-26T06:17:46.870219+0000","last_became_peered":"2024-07-26T06:17:46.870219+0000","last_unstale":"2024-07-26T06:17:46.880496+0000","last_undegraded":"2024-07-26T06:17:46.880496+0000","last_fullsized":"2024-07-26T06:17:46.880496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:27.562507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640055+0000","last_change":"2024-07-26T06:17:45.839166+0000","last_active":"2024-07-26T06:17:48.640055+0000","last_peered":"2024-07-26T06:17:48.640055+0000","last_clean":"2024-07-26T06:17:48.640055+0000","last_became_active":"2024-07-26T06:17:45.838866+0000","last_became_peered":"2024-07-26T06:17:45.838866+0000","last_unstale":"2024-07-26T06:17:48.640055+0000","last_undegraded":"2024-07-26T06:17:48.640055+0000","last_fullsized":"2024-07-26T06:17:48.640055+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:41:54.919363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.871752+0000","last_change":"2024-07-26T06:17:46.869327+0000","last_active":"2024-07-26T06:17:46.871752+0000","last_peered":"2024-07-26T06:17:46.871752+0000","last_clean":"2024-07-26T06:17:46.871752+0000","last_became_active":"2024-07-26T06:17:46.869010+0000","last_became_peered":"2024-07-26T06:17:46.869010+0000","last_unstale":"2024-07-26T06:17:46.871752+0000","last_undegraded":"2024-07-26T06:17:46.871752+0000","last_fullsized":"2024-07-26T06:17:46.871752+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:35.804674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.850350+0000","last_change":"2024-07-26T06:17:45.832174+0000","last_active":"2024-07-26T06:17:46.850350+0000","last_peered":"2024-07-26T06:17:46.850350+0000","last_clean":"2024-07-26T06:17:46.850350+0000","last_became_active":"2024-07-26T06:17:45.831879+0000","last_became_peered":"2024-07-26T06:17:45.831879+0000","last_unstale":"2024-07-26T06:17:46.850350+0000","last_undegraded":"2024-07-26T06:17:46.850350+0000","last_fullsized":"2024-07-26T06:17:46.850350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:37.746116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.874820+0000","last_change":"2024-07-26T06:17:46.872820+0000","last_active":"2024-07-26T06:17:46.874820+0000","last_peered":"2024-07-26T06:17:46.874820+0000","last_clean":"2024-07-26T06:17:46.874820+0000","last_became_active":"2024-07-26T06:17:46.872401+0000","last_became_peered":"2024-07-26T06:17:46.872401+0000","last_unstale":"2024-07-26T06:17:46.874820+0000","last_undegraded":"2024-07-26T06:17:46.874820+0000","last_fullsized":"2024-07-26T06:17:46.874820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:46.672362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142824+0000","last_change":"2024-07-26T06:17:45.831327+0000","last_active":"2024-07-26T06:17:48.142824+0000","last_peered":"2024-07-26T06:17:48.142824+0000","last_clean":"2024-07-26T06:17:48.142824+0000","last_became_active":"2024-07-26T06:17:45.831042+0000","last_became_peered":"2024-07-26T06:17:45.831042+0000","last_unstale":"2024-07-26T06:17:48.142824+0000","last_undegraded":"2024-07-26T06:17:48.142824+0000","last_fullsized":"2024-07-26T06:17:48.142824+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:45.821015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887270+0000","last_change":"2024-07-26T06:17:46.877535+0000","last_active":"2024-07-26T06:17:46.887270+0000","last_peered":"2024-07-26T06:17:46.887270+0000","last_clean":"2024-07-26T06:17:46.887270+0000","last_became_active":"2024-07-26T06:17:46.877254+0000","last_became_peered":"2024-07-26T06:17:46.877254+0000","last_unstale":"2024-07-26T06:17:46.887270+0000","last_undegraded":"2024-07-26T06:17:46.887270+0000","last_fullsized":"2024-07-26T06:17:46.887270+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:03.446774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870659+0000","last_change":"2024-07-26T06:17:45.849240+0000","last_active":"2024-07-26T06:17:46.870659+0000","last_peered":"2024-07-26T06:17:46.870659+0000","last_clean":"2024-07-26T06:17:46.870659+0000","last_became_active":"2024-07-26T06:17:45.848896+0000","last_became_peered":"2024-07-26T06:17:45.848896+0000","last_unstale":"2024-07-26T06:17:46.870659+0000","last_undegraded":"2024-07-26T06:17:46.870659+0000","last_fullsized":"2024-07-26T06:17:46.870659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:32.867527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885881+0000","last_change":"2024-07-26T06:17:46.876327+0000","last_active":"2024-07-26T06:17:46.885881+0000","last_peered":"2024-07-26T06:17:46.885881+0000","last_clean":"2024-07-26T06:17:46.885881+0000","last_became_active":"2024-07-26T06:17:46.876045+0000","last_became_peered":"2024-07-26T06:17:46.876045+0000","last_unstale":"2024-07-26T06:17:46.885881+0000","last_undegraded":"2024-07-26T06:17:46.885881+0000","last_fullsized":"2024-07-26T06:17:46.885881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:39:37.615124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.582469+0000","last_change":"2024-07-26T06:17:45.831283+0000","last_active":"2024-07-26T06:17:49.582469+0000","last_peered":"2024-07-26T06:17:49.582469+0000","last_clean":"2024-07-26T06:17:49.582469+0000","last_became_active":"2024-07-26T06:17:45.830822+0000","last_became_peered":"2024-07-26T06:17:45.830822+0000","last_unstale":"2024-07-26T06:17:49.582469+0000","last_undegraded":"2024-07-26T06:17:49.582469+0000","last_fullsized":"2024-07-26T06:17:49.582469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:17:09.948962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869815+0000","last_change":"2024-07-26T06:17:46.865363+0000","last_active":"2024-07-26T06:17:46.869815+0000","last_peered":"2024-07-26T06:17:46.869815+0000","last_clean":"2024-07-26T06:17:46.869815+0000","last_became_active":"2024-07-26T06:17:46.865083+0000","last_became_peered":"2024-07-26T06:17:46.865083+0000","last_unstale":"2024-07-26T06:17:46.869815+0000","last_undegraded":"2024-07-26T06:17:46.869815+0000","last_fullsized":"2024-07-26T06:17:46.869815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:00:30.072591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640766+0000","last_change":"2024-07-26T06:17:45.834795+0000","last_active":"2024-07-26T06:17:48.640766+0000","last_peered":"2024-07-26T06:17:48.640766+0000","last_clean":"2024-07-26T06:17:48.640766+0000","last_became_active":"2024-07-26T06:17:45.834387+0000","last_became_peered":"2024-07-26T06:17:45.834387+0000","last_unstale":"2024-07-26T06:17:48.640766+0000","last_undegraded":"2024-07-26T06:17:48.640766+0000","last_fullsized":"2024-07-26T06:17:48.640766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:25:26.176947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.875415+0000","last_change":"2024-07-26T06:17:46.867768+0000","last_active":"2024-07-26T06:17:46.875415+0000","last_peered":"2024-07-26T06:17:46.875415+0000","last_clean":"2024-07-26T06:17:46.875415+0000","last_became_active":"2024-07-26T06:17:46.867495+0000","last_became_peered":"2024-07-26T06:17:46.867495+0000","last_unstale":"2024-07-26T06:17:46.875415+0000","last_undegraded":"2024-07-26T06:17:46.875415+0000","last_fullsized":"2024-07-26T06:17:46.875415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:51:35.804674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640171+0000","last_change":"2024-07-26T06:17:45.842552+0000","last_active":"2024-07-26T06:17:48.640171+0000","last_peered":"2024-07-26T06:17:48.640171+0000","last_clean":"2024-07-26T06:17:48.640171+0000","last_became_active":"2024-07-26T06:17:45.842252+0000","last_became_peered":"2024-07-26T06:17:45.842252+0000","last_unstale":"2024-07-26T06:17:48.640171+0000","last_undegraded":"2024-07-26T06:17:48.640171+0000","last_fullsized":"2024-07-26T06:17:48.640171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:56.158056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870085+0000","last_change":"2024-07-26T06:17:46.867179+0000","last_active":"2024-07-26T06:17:46.870085+0000","last_peered":"2024-07-26T06:17:46.870085+0000","last_clean":"2024-07-26T06:17:46.870085+0000","last_became_active":"2024-07-26T06:17:46.866757+0000","last_became_peered":"2024-07-26T06:17:46.866757+0000","last_unstale":"2024-07-26T06:17:46.870085+0000","last_undegraded":"2024-07-26T06:17:46.870085+0000","last_fullsized":"2024-07-26T06:17:46.870085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:06.947898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870468+0000","last_change":"2024-07-26T06:17:45.847778+0000","last_active":"2024-07-26T06:17:46.870468+0000","last_peered":"2024-07-26T06:17:46.870468+0000","last_clean":"2024-07-26T06:17:46.870468+0000","last_became_active":"2024-07-26T06:17:45.846948+0000","last_became_peered":"2024-07-26T06:17:45.846948+0000","last_unstale":"2024-07-26T06:17:46.870468+0000","last_undegraded":"2024-07-26T06:17:46.870468+0000","last_fullsized":"2024-07-26T06:17:46.870468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:34:59.524300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880619+0000","last_change":"2024-07-26T06:17:46.875619+0000","last_active":"2024-07-26T06:17:46.880619+0000","last_peered":"2024-07-26T06:17:46.880619+0000","last_clean":"2024-07-26T06:17:46.880619+0000","last_became_active":"2024-07-26T06:17:46.875342+0000","last_became_peered":"2024-07-26T06:17:46.875342+0000","last_unstale":"2024-07-26T06:17:46.880619+0000","last_undegraded":"2024-07-26T06:17:46.880619+0000","last_fullsized":"2024-07-26T06:17:46.880619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:06.947898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.857240+0000","last_change":"2024-07-26T06:17:45.838946+0000","last_active":"2024-07-26T06:17:46.857240+0000","last_peered":"2024-07-26T06:17:46.857240+0000","last_clean":"2024-07-26T06:17:46.857240+0000","last_became_active":"2024-07-26T06:17:45.837548+0000","last_became_peered":"2024-07-26T06:17:45.837548+0000","last_unstale":"2024-07-26T06:17:46.857240+0000","last_undegraded":"2024-07-26T06:17:46.857240+0000","last_fullsized":"2024-07-26T06:17:46.857240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.120865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.864993+0000","last_change":"2024-07-26T06:17:46.861422+0000","last_active":"2024-07-26T06:17:46.864993+0000","last_peered":"2024-07-26T06:17:46.864993+0000","last_clean":"2024-07-26T06:17:46.864993+0000","last_became_active":"2024-07-26T06:17:46.861110+0000","last_became_peered":"2024-07-26T06:17:46.861110+0000","last_unstale":"2024-07-26T06:17:46.864993+0000","last_undegraded":"2024-07-26T06:17:46.864993+0000","last_fullsized":"2024-07-26T06:17:46.864993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:13:37.507234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.141917+0000","last_change":"2024-07-26T06:17:45.836257+0000","last_active":"2024-07-26T06:17:48.141917+0000","last_peered":"2024-07-26T06:17:48.141917+0000","last_clean":"2024-07-26T06:17:48.141917+0000","last_became_active":"2024-07-26T06:17:45.835965+0000","last_became_peered":"2024-07-26T06:17:45.835965+0000","last_unstale":"2024-07-26T06:17:48.141917+0000","last_undegraded":"2024-07-26T06:17:48.141917+0000","last_fullsized":"2024-07-26T06:17:48.141917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:44:30.764196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142718+0000","last_change":"2024-07-26T06:17:46.859786+0000","last_active":"2024-07-26T06:17:48.142718+0000","last_peered":"2024-07-26T06:17:48.142718+0000","last_clean":"2024-07-26T06:17:48.142718+0000","last_became_active":"2024-07-26T06:17:46.859401+0000","last_became_peered":"2024-07-26T06:17:46.859401+0000","last_unstale":"2024-07-26T06:17:48.142718+0000","last_undegraded":"2024-07-26T06:17:48.142718+0000","last_fullsized":"2024-07-26T06:17:48.142718+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:07.247741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885398+0000","last_change":"2024-07-26T06:17:45.845767+0000","last_active":"2024-07-26T06:17:46.885398+0000","last_peered":"2024-07-26T06:17:46.885398+0000","last_clean":"2024-07-26T06:17:46.885398+0000","last_became_active":"2024-07-26T06:17:45.845473+0000","last_became_peered":"2024-07-26T06:17:45.845473+0000","last_unstale":"2024-07-26T06:17:46.885398+0000","last_undegraded":"2024-07-26T06:17:46.885398+0000","last_fullsized":"2024-07-26T06:17:46.885398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:35:42.348603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.883735+0000","last_change":"2024-07-26T06:17:46.871870+0000","last_active":"2024-07-26T06:17:46.883735+0000","last_peered":"2024-07-26T06:17:46.883735+0000","last_clean":"2024-07-26T06:17:46.883735+0000","last_became_active":"2024-07-26T06:17:46.871559+0000","last_became_peered":"2024-07-26T06:17:46.871559+0000","last_unstale":"2024-07-26T06:17:46.883735+0000","last_undegraded":"2024-07-26T06:17:46.883735+0000","last_fullsized":"2024-07-26T06:17:46.883735+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:08:32.260066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.142870+0000","last_change":"2024-07-26T06:17:45.837138+0000","last_active":"2024-07-26T06:17:48.142870+0000","last_peered":"2024-07-26T06:17:48.142870+0000","last_clean":"2024-07-26T06:17:48.142870+0000","last_became_active":"2024-07-26T06:17:45.836830+0000","last_became_peered":"2024-07-26T06:17:45.836830+0000","last_unstale":"2024-07-26T06:17:48.142870+0000","last_undegraded":"2024-07-26T06:17:48.142870+0000","last_fullsized":"2024-07-26T06:17:48.142870+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:26:58.026700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640918+0000","last_change":"2024-07-26T06:17:46.864993+0000","last_active":"2024-07-26T06:17:48.640918+0000","last_peered":"2024-07-26T06:17:48.640918+0000","last_clean":"2024-07-26T06:17:48.640918+0000","last_became_active":"2024-07-26T06:17:46.861869+0000","last_became_peered":"2024-07-26T06:17:46.861869+0000","last_unstale":"2024-07-26T06:17:48.640918+0000","last_undegraded":"2024-07-26T06:17:48.640918+0000","last_fullsized":"2024-07-26T06:17:48.640918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:09:01.366872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887686+0000","last_change":"2024-07-26T06:17:45.845107+0000","last_active":"2024-07-26T06:17:46.887686+0000","last_peered":"2024-07-26T06:17:46.887686+0000","last_clean":"2024-07-26T06:17:46.887686+0000","last_became_active":"2024-07-26T06:17:45.844811+0000","last_became_peered":"2024-07-26T06:17:45.844811+0000","last_unstale":"2024-07-26T06:17:46.887686+0000","last_undegraded":"2024-07-26T06:17:46.887686+0000","last_fullsized":"2024-07-26T06:17:46.887686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:47:15.980042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.880437+0000","last_change":"2024-07-26T06:17:46.870832+0000","last_active":"2024-07-26T06:17:46.880437+0000","last_peered":"2024-07-26T06:17:46.880437+0000","last_clean":"2024-07-26T06:17:46.880437+0000","last_became_active":"2024-07-26T06:17:46.870484+0000","last_became_peered":"2024-07-26T06:17:46.870484+0000","last_unstale":"2024-07-26T06:17:46.880437+0000","last_undegraded":"2024-07-26T06:17:46.880437+0000","last_fullsized":"2024-07-26T06:17:46.880437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:17.279152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.860795+0000","last_change":"2024-07-26T06:17:45.842422+0000","last_active":"2024-07-26T06:17:46.860795+0000","last_peered":"2024-07-26T06:17:46.860795+0000","last_clean":"2024-07-26T06:17:46.860795+0000","last_became_active":"2024-07-26T06:17:45.841773+0000","last_became_peered":"2024-07-26T06:17:45.841773+0000","last_unstale":"2024-07-26T06:17:46.860795+0000","last_undegraded":"2024-07-26T06:17:46.860795+0000","last_fullsized":"2024-07-26T06:17:46.860795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:38:26.705105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.581753+0000","last_change":"2024-07-26T06:17:46.859692+0000","last_active":"2024-07-26T06:17:49.581753+0000","last_peered":"2024-07-26T06:17:49.581753+0000","last_clean":"2024-07-26T06:17:49.581753+0000","last_became_active":"2024-07-26T06:17:46.858742+0000","last_became_peered":"2024-07-26T06:17:46.858742+0000","last_unstale":"2024-07-26T06:17:49.581753+0000","last_undegraded":"2024-07-26T06:17:49.581753+0000","last_fullsized":"2024-07-26T06:17:49.581753+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:50.959089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.875536+0000","last_change":"2024-07-26T06:17:45.847558+0000","last_active":"2024-07-26T06:17:46.875536+0000","last_peered":"2024-07-26T06:17:46.875536+0000","last_clean":"2024-07-26T06:17:46.875536+0000","last_became_active":"2024-07-26T06:17:45.846183+0000","last_became_peered":"2024-07-26T06:17:45.846183+0000","last_unstale":"2024-07-26T06:17:46.875536+0000","last_undegraded":"2024-07-26T06:17:46.875536+0000","last_fullsized":"2024-07-26T06:17:46.875536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:48.622659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.581706+0000","last_change":"2024-07-26T06:17:46.876614+0000","last_active":"2024-07-26T06:17:49.581706+0000","last_peered":"2024-07-26T06:17:49.581706+0000","last_clean":"2024-07-26T06:17:49.581706+0000","last_became_active":"2024-07-26T06:17:46.876239+0000","last_became_peered":"2024-07-26T06:17:46.876239+0000","last_unstale":"2024-07-26T06:17:49.581706+0000","last_undegraded":"2024-07-26T06:17:49.581706+0000","last_fullsized":"2024-07-26T06:17:49.581706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:11.991070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.857429+0000","last_change":"2024-07-26T06:17:45.838684+0000","last_active":"2024-07-26T06:17:46.857429+0000","last_peered":"2024-07-26T06:17:46.857429+0000","last_clean":"2024-07-26T06:17:46.857429+0000","last_became_active":"2024-07-26T06:17:45.836740+0000","last_became_peered":"2024-07-26T06:17:45.836740+0000","last_unstale":"2024-07-26T06:17:46.857429+0000","last_undegraded":"2024-07-26T06:17:46.857429+0000","last_fullsized":"2024-07-26T06:17:46.857429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:13.612220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.581810+0000","last_change":"2024-07-26T06:17:46.877185+0000","last_active":"2024-07-26T06:17:49.581810+0000","last_peered":"2024-07-26T06:17:49.581810+0000","last_clean":"2024-07-26T06:17:49.581810+0000","last_became_active":"2024-07-26T06:17:46.876602+0000","last_became_peered":"2024-07-26T06:17:46.876602+0000","last_unstale":"2024-07-26T06:17:49.581810+0000","last_undegraded":"2024-07-26T06:17:49.581810+0000","last_fullsized":"2024-07-26T06:17:49.581810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:50.576582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.834886+0000","last_change":"2024-07-26T06:17:45.831813+0000","last_active":"2024-07-26T06:17:46.834886+0000","last_peered":"2024-07-26T06:17:46.834886+0000","last_clean":"2024-07-26T06:17:46.834886+0000","last_became_active":"2024-07-26T06:17:45.831532+0000","last_became_peered":"2024-07-26T06:17:45.831532+0000","last_unstale":"2024-07-26T06:17:46.834886+0000","last_undegraded":"2024-07-26T06:17:46.834886+0000","last_fullsized":"2024-07-26T06:17:46.834886+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.120865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.863895+0000","last_change":"2024-07-26T06:17:46.857651+0000","last_active":"2024-07-26T06:17:46.863895+0000","last_peered":"2024-07-26T06:17:46.863895+0000","last_clean":"2024-07-26T06:17:46.863895+0000","last_became_active":"2024-07-26T06:17:46.857133+0000","last_became_peered":"2024-07-26T06:17:46.857133+0000","last_unstale":"2024-07-26T06:17:46.863895+0000","last_undegraded":"2024-07-26T06:17:46.863895+0000","last_fullsized":"2024-07-26T06:17:46.863895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:35.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885309+0000","last_change":"2024-07-26T06:17:45.844087+0000","last_active":"2024-07-26T06:17:46.885309+0000","last_peered":"2024-07-26T06:17:46.885309+0000","last_clean":"2024-07-26T06:17:46.885309+0000","last_became_active":"2024-07-26T06:17:45.843797+0000","last_became_peered":"2024-07-26T06:17:45.843797+0000","last_unstale":"2024-07-26T06:17:46.885309+0000","last_undegraded":"2024-07-26T06:17:46.885309+0000","last_fullsized":"2024-07-26T06:17:46.885309+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:26:26.068716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.866058+0000","last_change":"2024-07-26T06:17:46.860008+0000","last_active":"2024-07-26T06:17:46.866058+0000","last_peered":"2024-07-26T06:17:46.866058+0000","last_clean":"2024-07-26T06:17:46.866058+0000","last_became_active":"2024-07-26T06:17:46.859731+0000","last_became_peered":"2024-07-26T06:17:46.859731+0000","last_unstale":"2024-07-26T06:17:46.866058+0000","last_undegraded":"2024-07-26T06:17:46.866058+0000","last_fullsized":"2024-07-26T06:17:46.866058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:28.166488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.857809+0000","last_change":"2024-07-26T06:17:45.837663+0000","last_active":"2024-07-26T06:17:46.857809+0000","last_peered":"2024-07-26T06:17:46.857809+0000","last_clean":"2024-07-26T06:17:46.857809+0000","last_became_active":"2024-07-26T06:17:45.837276+0000","last_became_peered":"2024-07-26T06:17:45.837276+0000","last_unstale":"2024-07-26T06:17:46.857809+0000","last_undegraded":"2024-07-26T06:17:46.857809+0000","last_fullsized":"2024-07-26T06:17:46.857809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:18.959935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.887129+0000","last_change":"2024-07-26T06:17:46.880623+0000","last_active":"2024-07-26T06:17:46.887129+0000","last_peered":"2024-07-26T06:17:46.887129+0000","last_clean":"2024-07-26T06:17:46.887129+0000","last_became_active":"2024-07-26T06:17:46.880329+0000","last_became_peered":"2024-07-26T06:17:46.880329+0000","last_unstale":"2024-07-26T06:17:46.887129+0000","last_undegraded":"2024-07-26T06:17:46.887129+0000","last_fullsized":"2024-07-26T06:17:46.887129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:54:24.367302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.854853+0000","last_change":"2024-07-26T06:17:45.825394+0000","last_active":"2024-07-26T06:17:46.854853+0000","last_peered":"2024-07-26T06:17:46.854853+0000","last_clean":"2024-07-26T06:17:46.854853+0000","last_became_active":"2024-07-26T06:17:45.824922+0000","last_became_peered":"2024-07-26T06:17:45.824922+0000","last_unstale":"2024-07-26T06:17:46.854853+0000","last_undegraded":"2024-07-26T06:17:46.854853+0000","last_fullsized":"2024-07-26T06:17:46.854853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:01.680799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.883348+0000","last_change":"2024-07-26T06:17:46.871923+0000","last_active":"2024-07-26T06:17:46.883348+0000","last_peered":"2024-07-26T06:17:46.883348+0000","last_clean":"2024-07-26T06:17:46.883348+0000","last_became_active":"2024-07-26T06:17:46.871597+0000","last_became_peered":"2024-07-26T06:17:46.871597+0000","last_unstale":"2024-07-26T06:17:46.883348+0000","last_undegraded":"2024-07-26T06:17:46.883348+0000","last_fullsized":"2024-07-26T06:17:46.883348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:46.672362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.835072+0000","last_change":"2024-07-26T06:17:45.839615+0000","last_active":"2024-07-26T06:17:46.835072+0000","last_peered":"2024-07-26T06:17:46.835072+0000","last_clean":"2024-07-26T06:17:46.835072+0000","last_became_active":"2024-07-26T06:17:45.839286+0000","last_became_peered":"2024-07-26T06:17:45.839286+0000","last_unstale":"2024-07-26T06:17:46.835072+0000","last_undegraded":"2024-07-26T06:17:46.835072+0000","last_fullsized":"2024-07-26T06:17:46.835072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:13.612220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.869558+0000","last_change":"2024-07-26T06:17:46.864454+0000","last_active":"2024-07-26T06:17:46.869558+0000","last_peered":"2024-07-26T06:17:46.869558+0000","last_clean":"2024-07-26T06:17:46.869558+0000","last_became_active":"2024-07-26T06:17:46.864188+0000","last_became_peered":"2024-07-26T06:17:46.864188+0000","last_unstale":"2024-07-26T06:17:46.869558+0000","last_undegraded":"2024-07-26T06:17:46.869558+0000","last_fullsized":"2024-07-26T06:17:46.869558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:05:31.790582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.853827+0000","last_change":"2024-07-26T06:17:45.837315+0000","last_active":"2024-07-26T06:17:46.853827+0000","last_peered":"2024-07-26T06:17:46.853827+0000","last_clean":"2024-07-26T06:17:46.853827+0000","last_became_active":"2024-07-26T06:17:45.836993+0000","last_became_peered":"2024-07-26T06:17:45.836993+0000","last_unstale":"2024-07-26T06:17:46.853827+0000","last_undegraded":"2024-07-26T06:17:46.853827+0000","last_fullsized":"2024-07-26T06:17:46.853827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:26:18.248918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885819+0000","last_change":"2024-07-26T06:17:46.877325+0000","last_active":"2024-07-26T06:17:46.885819+0000","last_peered":"2024-07-26T06:17:46.885819+0000","last_clean":"2024-07-26T06:17:46.885819+0000","last_became_active":"2024-07-26T06:17:46.877004+0000","last_became_peered":"2024-07-26T06:17:46.877004+0000","last_unstale":"2024-07-26T06:17:46.885819+0000","last_undegraded":"2024-07-26T06:17:46.885819+0000","last_fullsized":"2024-07-26T06:17:46.885819+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:47:58.687274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.639834+0000","last_change":"2024-07-26T06:17:45.838377+0000","last_active":"2024-07-26T06:17:48.639834+0000","last_peered":"2024-07-26T06:17:48.639834+0000","last_clean":"2024-07-26T06:17:48.639834+0000","last_became_active":"2024-07-26T06:17:45.838098+0000","last_became_peered":"2024-07-26T06:17:45.838098+0000","last_unstale":"2024-07-26T06:17:48.639834+0000","last_undegraded":"2024-07-26T06:17:48.639834+0000","last_fullsized":"2024-07-26T06:17:48.639834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:15:28.968555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.640298+0000","last_change":"2024-07-26T06:17:46.860472+0000","last_active":"2024-07-26T06:17:48.640298+0000","last_peered":"2024-07-26T06:17:48.640298+0000","last_clean":"2024-07-26T06:17:48.640298+0000","last_became_active":"2024-07-26T06:17:46.860213+0000","last_became_peered":"2024-07-26T06:17:46.860213+0000","last_unstale":"2024-07-26T06:17:48.640298+0000","last_undegraded":"2024-07-26T06:17:48.640298+0000","last_fullsized":"2024-07-26T06:17:48.640298+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:18.444000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:48.141709+0000","last_change":"2024-07-26T06:17:45.840975+0000","last_active":"2024-07-26T06:17:48.141709+0000","last_peered":"2024-07-26T06:17:48.141709+0000","last_clean":"2024-07-26T06:17:48.141709+0000","last_became_active":"2024-07-26T06:17:45.840717+0000","last_became_peered":"2024-07-26T06:17:45.840717+0000","last_unstale":"2024-07-26T06:17:48.141709+0000","last_undegraded":"2024-07-26T06:17:48.141709+0000","last_fullsized":"2024-07-26T06:17:48.141709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:09:00.361912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.870138+0000","last_change":"2024-07-26T06:17:46.862600+0000","last_active":"2024-07-26T06:17:46.870138+0000","last_peered":"2024-07-26T06:17:46.870138+0000","last_clean":"2024-07-26T06:17:46.870138+0000","last_became_active":"2024-07-26T06:17:46.862055+0000","last_became_peered":"2024-07-26T06:17:46.862055+0000","last_unstale":"2024-07-26T06:17:46.870138+0000","last_undegraded":"2024-07-26T06:17:46.870138+0000","last_fullsized":"2024-07-26T06:17:46.870138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:54:38.354576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.871666+0000","last_change":"2024-07-26T06:17:45.840005+0000","last_active":"2024-07-26T06:17:46.871666+0000","last_peered":"2024-07-26T06:17:46.871666+0000","last_clean":"2024-07-26T06:17:46.871666+0000","last_became_active":"2024-07-26T06:17:45.839706+0000","last_became_peered":"2024-07-26T06:17:45.839706+0000","last_unstale":"2024-07-26T06:17:46.871666+0000","last_undegraded":"2024-07-26T06:17:46.871666+0000","last_fullsized":"2024-07-26T06:17:46.871666+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:22:45.516586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-07-26T06:17:49.582011+0000","last_change":"2024-07-26T06:17:46.878196+0000","last_active":"2024-07-26T06:17:49.582011+0000","last_peered":"2024-07-26T06:17:49.582011+0000","last_clean":"2024-07-26T06:17:49.582011+0000","last_became_active":"2024-07-26T06:17:46.877410+0000","last_became_peered":"2024-07-26T06:17:46.877410+0000","last_unstale":"2024-07-26T06:17:49.582011+0000","last_undegraded":"2024-07-26T06:17:49.582011+0000","last_fullsized":"2024-07-26T06:17:49.582011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:45.798295+0000","last_clean_scrub_stamp":"2024-07-26T06:17:45.798295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:15.812268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.885644+0000","last_change":"2024-07-26T06:17:45.847226+0000","last_active":"2024-07-26T06:17:46.885644+0000","last_peered":"2024-07-26T06:17:46.885644+0000","last_clean":"2024-07-26T06:17:46.885644+0000","last_became_active":"2024-07-26T06:17:45.846508+0000","last_became_peered":"2024-07-26T06:17:45.846508+0000","last_unstale":"2024-07-26T06:17:46.885644+0000","last_undegraded":"2024-07-26T06:17:46.885644+0000","last_fullsized":"2024-07-26T06:17:46.885644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:17:44.793335+0000","last_clean_scrub_stamp":"2024-07-26T06:17:44.793335+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:32:10.105503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":196,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-07-26T06:17:46.853866+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:17:46.853866+0000","last_peered":"2024-07-26T06:17:46.853866+0000","last_clean":"2024-07-26T06:17:46.853866+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:17:46.853866+0000","last_undegraded":"2024-07-26T06:17:46.853866+0000","last_fullsized":"2024-07-26T06:17:46.853866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":933939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":336180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509496,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":823296,"data_stored":343045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":344567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1413120,"data_stored":933323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542201,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":349325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542200,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":839680,"data_stored":349367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":351483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:17:51.794 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:17:51.794 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.052+0000 7f66f0c00640 1 -- 172.21.15.163:0/4045514972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0003a30 msgr2=0x7f66d0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.052+0000 7f66f0c00640 1 --2- 172.21.15.163:0/4045514972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66d0003a30 0x7f66d0003e50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00c2e0 tx=0x7f66d0005830 comp rx=0 tx=0).stop 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.052+0000 7f66ebe00640 1 -- 172.21.15.163:0/4045514972 reap_dead start 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.053+0000 7f66f0c00640 1 -- 172.21.15.163:0/4045514972 shutdown_connections 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.053+0000 7f66f0c00640 1 -- 172.21.15.163:0/4045514972 >> 172.21.15.163:0/4045514972 conn(0x7f66ec0b0e30 msgr2=0x7f66ec0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.053+0000 7f66f0c00640 1 -- 172.21.15.163:0/4045514972 shutdown_connections 2024-07-26T06:17:52.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.053+0000 7f66f0c00640 1 -- 172.21.15.163:0/4045514972 wait complete. 2024-07-26T06:17:52.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.054+0000 7f66f0c00640 1 Processor -- start 2024-07-26T06:17:52.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.054+0000 7f66f0c00640 1 -- start start 2024-07-26T06:17:52.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.055+0000 7f66f0c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66d0003a30 0x7f66ec149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.055+0000 7f66f0c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66ec152ba0 0x7f66ec14c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.055+0000 7f66f0c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 0x7f66ec0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.055+0000 7f66f0c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec164310 con 0x7f66ec0ccf10 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.055+0000 7f66f0c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec164190 con 0x7f66d0003a30 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.055+0000 7f66f0c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f66ec164490 con 0x7f66ec152ba0 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 0x7f66ec0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 0x7f66ec0cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34218/0 (socket says 172.21.15.163:34218) 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 -- 172.21.15.163:0/1660446538 learned_addr learned my addr 172.21.15.163:0/1660446538 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66eb400640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66d0003a30 0x7f66ec149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:52.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 -- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66ec152ba0 msgr2=0x7f66ec14c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66ec152ba0 0x7f66ec14c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 -- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66d0003a30 msgr2=0x7f66ec149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66d0003a30 0x7f66ec149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 -- 172.21.15.163:0/1660446538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec14a410 con 0x7f66ec0ccf10 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66eb400640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66d0003a30 0x7f66ec149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.056+0000 7f66ebe00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 0x7f66ec0cd390 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f66e4005fd0 tx=0x7f66e400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.057+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e400cc00 con 0x7f66ec0ccf10 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.057+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66e4019070 con 0x7f66ec0ccf10 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.057+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e40116d0 con 0x7f66ec0ccf10 2024-07-26T06:17:52.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.057+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec14c870 con 0x7f66ec0ccf10 2024-07-26T06:17:52.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.057+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec1530c0 con 0x7f66ec0ccf10 2024-07-26T06:17:52.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.058+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec0afe20 con 0x7f66ec0ccf10 2024-07-26T06:17:52.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.059+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f66e4011870 con 0x7f66ec0ccf10 2024-07-26T06:17:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.060+0000 7f66e3400640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f66c8076630 0x7f66c8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.060+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f66e4098220 con 0x7f66ec0ccf10 2024-07-26T06:17:52.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.061+0000 7f66eb400640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f66c8076630 0x7f66c8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:52.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.062+0000 7f66eb400640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f66c8076630 0x7f66c8078af0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f66dc033000 tx=0x7f66dc001a80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:52.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.064+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f66ec14a410 con 0x7f66ec0ccf10 2024-07-26T06:17:52.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.350+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f66ec15b740 con 0x7f66ec0ccf10 2024-07-26T06:17:52.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.354+0000 7f66e3400640 1 -- 172.21.15.163:0/1660446538 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f66ec15b740 con 0x7f66ec0ccf10 2024-07-26T06:17:52.355 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:17:52.355 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:17:52.355 INFO:teuthology.orchestra.run.smithi163.stdout: caps mds = "allow" 2024-07-26T06:17:52.355 INFO:teuthology.orchestra.run.smithi163.stdout: caps mon = "allow r" 2024-07-26T06:17:52.355 INFO:teuthology.orchestra.run.smithi163.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-07-26T06:17:52.355 INFO:teuthology.orchestra.run.smithi163.stderr:updated caps for client.0 2024-07-26T06:17:52.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.356+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f66c8076630 msgr2=0x7f66c8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.356+0000 7f66f0c00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f66c8076630 0x7f66c8078af0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f66dc033000 tx=0x7f66dc001a80 comp rx=0 tx=0).stop 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.356+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 msgr2=0x7f66ec0cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.356+0000 7f66f0c00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 0x7f66ec0cd390 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f66e4005fd0 tx=0x7f66e400eda0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 shutdown_connections 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f66c8076630 0x7f66c8078af0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66ec0ccf10 0x7f66ec0cd390 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66ec152ba0 0x7f66ec14c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 --2- 172.21.15.163:0/1660446538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66d0003a30 0x7f66ec149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 >> 172.21.15.163:0/1660446538 conn(0x7f66ec0b0e30 msgr2=0x7f66ec15a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 shutdown_connections 2024-07-26T06:17:52.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.357+0000 7f66f0c00640 1 -- 172.21.15.163:0/1660446538 wait complete. 2024-07-26T06:17:52.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:17:52.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942de00640 1 -- 172.21.15.163:0/687302898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 msgr2=0x7f940c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942de00640 1 --2- 172.21.15.163:0/687302898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f940c0051c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f942000bb20 tx=0x7f940c005830 comp rx=0 tx=0).stop 2024-07-26T06:17:52.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942d400640 1 -- 172.21.15.163:0/687302898 reap_dead start 2024-07-26T06:17:52.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942de00640 1 -- 172.21.15.163:0/687302898 shutdown_connections 2024-07-26T06:17:52.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942de00640 1 -- 172.21.15.163:0/687302898 >> 172.21.15.163:0/687302898 conn(0x7f94280b0e30 msgr2=0x7f94280b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:52.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942de00640 1 -- 172.21.15.163:0/687302898 shutdown_connections 2024-07-26T06:17:52.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.561+0000 7f942de00640 1 -- 172.21.15.163:0/687302898 wait complete. 2024-07-26T06:17:52.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.563+0000 7f942de00640 1 Processor -- start 2024-07-26T06:17:52.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.563+0000 7f942de00640 1 -- start start 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f942813aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942de00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94280ab480 0x7f942814d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942de00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f942814dcd0 0x7f94280a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942de00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f94280b3aa0 con 0x7f940c004d40 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942de00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f94280b3920 con 0x7f942814dcd0 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942de00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f94280b3c20 con 0x7f94280ab480 2024-07-26T06:17:52.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f942813aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:52.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f942813aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34230/0 (socket says 172.21.15.163:34230) 2024-07-26T06:17:52.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.564+0000 7f942ca00640 1 -- 172.21.15.163:0/285221994 learned_addr learned my addr 172.21.15.163:0/285221994 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942d400640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f942814dcd0 0x7f94280a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942ca00640 1 -- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94280ab480 msgr2=0x7f942814d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942ca00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94280ab480 0x7f942814d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942ca00640 1 -- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f942814dcd0 msgr2=0x7f94280a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942ca00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f942814dcd0 0x7f94280a94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942ca00640 1 -- 172.21.15.163:0/285221994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942814e250 con 0x7f940c004d40 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942ca00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f942813aa20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f941800ebf0 tx=0x7f941800c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418004b20 con 0x7f940c004d40 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9418004db0 con 0x7f940c004d40 2024-07-26T06:17:52.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418005280 con 0x7f940c004d40 2024-07-26T06:17:52.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942814e530 con 0x7f940c004d40 2024-07-26T06:17:52.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.565+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942814eaf0 con 0x7f940c004d40 2024-07-26T06:17:52.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.567+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942806fb10 con 0x7f940c004d40 2024-07-26T06:17:52.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.568+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f9418005420 con 0x7f940c004d40 2024-07-26T06:17:52.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.569+0000 7f9425600640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9404076630 0x7f9404078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:52.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.569+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f9418098a20 con 0x7f940c004d40 2024-07-26T06:17:52.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.569+0000 7f941fe00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9404076630 0x7f9404078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:52.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.570+0000 7f941fe00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9404076630 0x7f9404078af0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9428154960 tx=0x7f9410001680 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:52.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.573+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f942806fb10 con 0x7f940c004d40 2024-07-26T06:17:52.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.799+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f942815b0c0 con 0x7f940c004d40 2024-07-26T06:17:52.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.800+0000 7f9425600640 1 -- 172.21.15.163:0/285221994 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f9418016030 con 0x7f940c004d40 2024-07-26T06:17:52.801 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:17:52.801 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":45,"btime":"2024-07-26T06:17:49:877600+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:17:52.801 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 45 2024-07-26T06:17:52.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9404076630 msgr2=0x7f9404078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9404076630 0x7f9404078af0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9428154960 tx=0x7f9410001680 comp rx=0 tx=0).stop 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 msgr2=0x7f942813aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f942813aa20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f941800ebf0 tx=0x7f941800c640 comp rx=0 tx=0).stop 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 shutdown_connections 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f9404076630 0x7f9404078af0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f942814dcd0 0x7f94280a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94280ab480 0x7f942814d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 --2- 172.21.15.163:0/285221994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f940c004d40 0x7f942813aa20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:52.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 >> 172.21.15.163:0/285221994 conn(0x7f94280b0e30 msgr2=0x7f942815df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:52.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 shutdown_connections 2024-07-26T06:17:52.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:52.803+0000 7f942de00640 1 -- 172.21.15.163:0/285221994 wait complete. 2024-07-26T06:17:53.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:17:46.285783+0000', 'modified': '2024-07-26T06:17:49.380437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6090}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6090': {'gid': 6090, 'name': 'c', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.163:6835/1648150930', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6833', 'nonce': 1648150930}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 1648150930}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6090, 'qdb_cluster': [6090]} 2024-07-26T06:17:53.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:17:53.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-07-26T06:17:53.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 -- 172.21.15.163:0/2938507975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 msgr2=0x7f45d415ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 --2- 172.21.15.163:0/2938507975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d415ec00 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f45d406ce80 tx=0x7f45cc02aa10 comp rx=0 tx=0).stop 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 -- 172.21.15.163:0/2938507975 shutdown_connections 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 --2- 172.21.15.163:0/2938507975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d415ec00 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 --2- 172.21.15.163:0/2938507975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d40ce2d0 0x7f45d4071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 --2- 172.21.15.163:0/2938507975 >> v2:172.21.15.164:3300/0 conn(0x7f45d41556f0 0x7f45d40cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 -- 172.21.15.163:0/2938507975 >> 172.21.15.163:0/2938507975 conn(0x7f45d40b0e30 msgr2=0x7f45d40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 -- 172.21.15.163:0/2938507975 shutdown_connections 2024-07-26T06:17:53.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.441+0000 7f45d3200640 1 -- 172.21.15.163:0/2938507975 wait complete. 2024-07-26T06:17:53.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.443+0000 7f45d3200640 1 Processor -- start 2024-07-26T06:17:53.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.443+0000 7f45d3200640 1 -- start start 2024-07-26T06:17:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d41477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d3200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d40ce2d0 0x7f45d415c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d3200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d41556f0 0x7f45d415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d3200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45d415f2c0 con 0x7f45d4071a70 2024-07-26T06:17:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d3200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45d415f140 con 0x7f45d41556f0 2024-07-26T06:17:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d3200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45d415f440 con 0x7f45d40ce2d0 2024-07-26T06:17:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.444+0000 7f45d1e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d41477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d41477c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34240/0 (socket says 172.21.15.163:34240) 2024-07-26T06:17:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 -- 172.21.15.163:0/4055458052 learned_addr learned my addr 172.21.15.163:0/4055458052 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45cbe00640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d40ce2d0 0x7f45d415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 -- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d40ce2d0 msgr2=0x7f45d415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d2800640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d41556f0 0x7f45d415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d40ce2d0 0x7f45d415c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 -- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d41556f0 msgr2=0x7f45d415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d41556f0 0x7f45d415ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 -- 172.21.15.163:0/4055458052 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d414d5a0 con 0x7f45d4071a70 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.445+0000 7f45d1e00640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d41477c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c9a0 tx=0x7f45c400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.446+0000 7f45ca000640 1 -- 172.21.15.163:0/4055458052 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4007bb0 con 0x7f45d4071a70 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.446+0000 7f45ca000640 1 -- 172.21.15.163:0/4055458052 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45c4007e40 con 0x7f45d4071a70 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.446+0000 7f45ca000640 1 -- 172.21.15.163:0/4055458052 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400f690 con 0x7f45d4071a70 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.446+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d4152590 con 0x7f45d4071a70 2024-07-26T06:17:53.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.446+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d4152b50 con 0x7f45d4071a70 2024-07-26T06:17:53.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.448+0000 7f45ca000640 1 -- 172.21.15.163:0/4055458052 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f45c400f830 con 0x7f45d4071a70 2024-07-26T06:17:53.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.450+0000 7f45ca000640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f45b0076630 0x7f45b0078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.450+0000 7f45ca000640 1 -- 172.21.15.163:0/4055458052 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f45c40984d0 con 0x7f45d4071a70 2024-07-26T06:17:53.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.450+0000 7f45cbe00640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f45b0076630 0x7f45b0078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.450+0000 7f45db7c5740 1 Processor -- start 2024-07-26T06:17:53.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.451+0000 7f45db7c5740 1 -- start start 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ff940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3fe8d0 0x55df4c3fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 0x55df4c3b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45db7c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55df4c2cf3f0 con 0x55df4c3b9e50 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55df4c32a5c0 con 0x55df4c3ff560 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55df4c319b60 con 0x55df4c3fe8d0 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45cbe00640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f45b0076630 0x7f45b0078af0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f45d415d280 tx=0x7f45bc005c90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 0x55df4c3b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 0x55df4c3b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34242/0 (socket says 172.21.15.163:34242) 2024-07-26T06:17:53.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 -- 172.21.15.163:0/3588379585 learned_addr learned my addr 172.21.15.163:0/3588379585 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45cbe00640 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3fe8d0 0x55df4c3fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d1e00640 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 -- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3fe8d0 msgr2=0x55df4c3fcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3fe8d0 0x55df4c3fcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 -- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 msgr2=0x55df4c3ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d2800640 1 -- 172.21.15.163:0/3588379585 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df4c07e850 con 0x55df4c3b9e50 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.452+0000 7f45d1e00640 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.453+0000 7f45d2800640 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 0x55df4c3b2fd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f45cc02b000 tx=0x7f45cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e78828e99a66240 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.453+0000 7f45c2600640 1 -- 172.21.15.163:0/3588379585 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc033070 con 0x55df4c3b9e50 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.453+0000 7f45c2600640 1 -- 172.21.15.163:0/3588379585 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45cc003a70 con 0x55df4c3b9e50 2024-07-26T06:17:53.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.453+0000 7f45c2600640 1 -- 172.21.15.163:0/3588379585 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003e60 con 0x55df4c3b9e50 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 -- 172.21.15.163:0/3588379585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 msgr2=0x55df4c3b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 0x55df4c3b2fd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f45cc02b000 tx=0x7f45cc004a40 comp rx=0 tx=0).stop 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 -- 172.21.15.163:0/3588379585 shutdown_connections 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b9e50 0x55df4c3b2fd0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3fe8d0 0x55df4c3fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3588379585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 -- 172.21.15.163:0/3588379585 >> 172.21.15.163:0/3588379585 conn(0x55df4c3b51a0 msgr2=0x55df4c3fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 -- 172.21.15.163:0/3588379585 shutdown_connections 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.454+0000 7f45db7c5740 1 -- 172.21.15.163:0/3588379585 wait complete. 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.455+0000 7f45db7c5740 1 -- start start 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.455+0000 7f45db7c5740 1 Processor -- start 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ddcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c410aa0 0x55df4c3b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 0x55df4c3e2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55df4c2c2210 con 0x55df4c3b3990 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55df4c31dba0 con 0x55df4c3ff560 2024-07-26T06:17:53.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55df4c3fdce0 con 0x55df4c410aa0 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45db7c5740 10 client.0 _ll_register_callbacks cb 0x55df4c2c28a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 0x55df4c3e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 0x55df4c3e2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34258/0 (socket says 172.21.15.163:34258) 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45d2800640 1 -- 172.21.15.163:0/462114126 learned_addr learned my addr 172.21.15.163:0/462114126 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45cbe00640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c410aa0 0x55df4c3b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45d2800640 1 -- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c410aa0 msgr2=0x55df4c3b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45d1e00640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ddcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.456+0000 7f45d2800640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c410aa0 0x55df4c3b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45d2800640 1 -- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 msgr2=0x55df4c3ddcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45d2800640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ddcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45d2800640 1 -- 172.21.15.163:0/462114126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df4c0f56b0 con 0x55df4c3b3990 2024-07-26T06:17:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45d2800640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 0x55df4c3e2640 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f45cc005fd0 tx=0x7f45cc02fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24dc48d060f0253b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:17:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45b6a00640 1 -- 172.21.15.163:0/462114126 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc033070 con 0x55df4c3b3990 2024-07-26T06:17:53.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45b6a00640 1 -- 172.21.15.163:0/462114126 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f45cc038660 con 0x55df4c3b3990 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55df4c32c510 con 0x55df4c3b3990 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45b6a00640 1 -- 172.21.15.163:0/462114126 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f45cc038860 con 0x55df4c3b3990 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45db7c5740 10 client.6225 fetch_fsmap learned FSMap version 45 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.457+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55df4c3dec60 con 0x55df4c3b3990 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45b6a00640 1 -- 172.21.15.163:0/462114126 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55df4c32c510 con 0x55df4c3b3990 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45b6a00640 1 -- 172.21.15.163:0/462114126 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 45) ==== 3502+0+0 (secure 0 0 0) 0x7f45cc038a00 con 0x55df4c3b3990 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45db7c5740 10 client.6225 fetch_fsmap finished waiting for FSMap version 45 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45db7c5740 10 client.6225 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45db7c5740 20 client.6225 populate_metadata read hostname 'smithi163' 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45db7c5740 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c404e30 0x55df4c405210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45db7c5740 4 client.6225 mds_command: new command op to 6090 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 --> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df4c327fb0 con 0x55df4c404e30 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.458+0000 7f45d1e00640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c404e30 0x55df4c405210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.459+0000 7f45d1e00640 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c404e30 0x55df4c405210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fffe8adb40cbbf6 server_cookie=d7e5acbaefcb0d00 in_seq=0 out_seq=0 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.459+0000 7f45b6a00640 10 client.6225 ms_handle_connect on v2:172.21.15.163:6833/1648150930 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.459+0000 7f45b6a00640 1 -- 172.21.15.163:0/462114126 <== mds.0 v2:172.21.15.163:6833/1648150930 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55df4c327fb0 con 0x55df4c404e30 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.459+0000 7f45b6a00640 10 client.6225 handle_command_reply: tid=0 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 client.6225 shutdown 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 msgr2=0x55df4c3e2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 0x55df4c3e2640 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f45cc005fd0 tx=0x7f45cc02fdc0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 shutdown_connections 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c404e30 0x55df4c405210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c3b3990 0x55df4c3e2640 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c410aa0 0x55df4c3b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 --2- 172.21.15.163:0/462114126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3ddcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45b6a00640 0 client.6225 ms_handle_reset on v2:172.21.15.163:6833/1648150930 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 >> 172.21.15.163:0/462114126 conn(0x55df4c3b9e50 msgr2=0x55df4c3b8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 shutdown_connections 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 1 -- 172.21.15.163:0/462114126 wait complete. 2024-07-26T06:17:53.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.460+0000 7f45db7c5740 20 client.6225 trim_cache size 0 max 16384 2024-07-26T06:17:53.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.480+0000 7f45db7c5740 1 Processor -- start 2024-07-26T06:17:53.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 -- start start 2024-07-26T06:17:53.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3b3990 0x55df4c3fcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3ff560 0x55df4c3e2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c3b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ff3e0 con 0x55df4c410aa0 2024-07-26T06:17:53.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45d414ecb0 con 0x55df4c3b3990 2024-07-26T06:17:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.481+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45d41fede0 con 0x55df4c3ff560 2024-07-26T06:17:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.482+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c3b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.482+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c3b7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34266/0 (socket says 172.21.15.163:34266) 2024-07-26T06:17:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.482+0000 7f45d2800640 1 -- 172.21.15.163:0/3244961534 learned_addr learned my addr 172.21.15.163:0/3244961534 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.482+0000 7f45cbe00640 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3ff560 0x55df4c3e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.482+0000 7f45d1e00640 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3b3990 0x55df4c3fcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45d2800640 1 -- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3ff560 msgr2=0x55df4c3e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45d2800640 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3ff560 0x55df4c3e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45d2800640 1 -- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3b3990 msgr2=0x55df4c3fcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45d2800640 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3b3990 0x55df4c3fcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45d2800640 1 -- 172.21.15.163:0/3244961534 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df4c3fb180 con 0x55df4c410aa0 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45cbe00640 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3ff560 0x55df4c3e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45d2800640 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c3b7fb0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f45cc002230 tx=0x7f45cc02af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831a8e40fc7ab3d0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45c1c00640 1 -- 172.21.15.163:0/3244961534 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc033040 con 0x55df4c410aa0 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45c1c00640 1 -- 172.21.15.163:0/3244961534 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45cc038660 con 0x55df4c410aa0 2024-07-26T06:17:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.483+0000 7f45c1c00640 1 -- 172.21.15.163:0/3244961534 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc04b450 con 0x55df4c410aa0 2024-07-26T06:17:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 -- 172.21.15.163:0/3244961534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 msgr2=0x55df4c3b7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c3b7fb0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f45cc002230 tx=0x7f45cc02af10 comp rx=0 tx=0).stop 2024-07-26T06:17:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 -- 172.21.15.163:0/3244961534 shutdown_connections 2024-07-26T06:17:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c3b7fb0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3ff560 0x55df4c3e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3244961534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3b3990 0x55df4c3fcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 -- 172.21.15.163:0/3244961534 >> 172.21.15.163:0/3244961534 conn(0x55df4c3b9e50 msgr2=0x55df4c40fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 -- 172.21.15.163:0/3244961534 shutdown_connections 2024-07-26T06:17:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.484+0000 7f45db7c5740 1 -- 172.21.15.163:0/3244961534 wait complete. 2024-07-26T06:17:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.485+0000 7f45db7c5740 1 -- start start 2024-07-26T06:17:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.485+0000 7f45db7c5740 1 Processor -- start 2024-07-26T06:17:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3b3990 0x55df4c47b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3e2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c4211c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55df4c2c2210 con 0x55df4c410aa0 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55df4c31dba0 con 0x55df4c3ff560 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55df4c3de230 con 0x55df4c3b3990 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45db7c5740 10 client.0 _ll_register_callbacks cb 0x7f45cc02fa50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c4211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45d2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c4211c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34270/0 (socket says 172.21.15.163:34270) 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.486+0000 7f45d2800640 1 -- 172.21.15.163:0/3930077096 learned_addr learned my addr 172.21.15.163:0/3930077096 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d1e00640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3b3990 0x55df4c47b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45cbe00640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3e2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d2800640 1 -- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3b3990 msgr2=0x55df4c47b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d2800640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3b3990 0x55df4c47b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d2800640 1 -- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 msgr2=0x55df4c3e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:17:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d2800640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3e2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d2800640 1 -- 172.21.15.163:0/3930077096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df4c3e2970 con 0x55df4c410aa0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45d2800640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c4211c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f45cc002230 tx=0x7f45cc04b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a07fcb23ca787f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.487+0000 7f45b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45b6a00640 1 -- 172.21.15.163:0/3930077096 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc057030 con 0x55df4c410aa0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45b6a00640 1 -- 172.21.15.163:0/3930077096 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f45cc03e070 con 0x55df4c410aa0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45cc038660 con 0x55df4c410aa0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45b6a00640 1 -- 172.21.15.163:0/3930077096 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f45cc063050 con 0x55df4c410aa0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 10 client.6231 fetch_fsmap learned FSMap version 45 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55df4c4165b0 con 0x55df4c410aa0 2024-07-26T06:17:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45b6a00640 1 -- 172.21.15.163:0/3930077096 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45cc038660 con 0x55df4c410aa0 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45b6a00640 1 -- 172.21.15.163:0/3930077096 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 45) ==== 3502+0+0 (secure 0 0 0) 0x7f45cc066020 con 0x55df4c410aa0 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 10 client.6231 fetch_fsmap finished waiting for FSMap version 45 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 10 client.6231 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 20 client.6231 populate_metadata read hostname 'smithi163' 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c420580 0x55df4c420960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 4 client.6231 mds_command: new command op to 6090 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.488+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 --> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] -- command(tid 0: {"prefix": "status"}) -- 0x55df4c3256f0 con 0x55df4c420580 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45d1e00640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c420580 0x55df4c420960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45d1e00640 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c420580 0x55df4c420960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=300b86e7c95e6ba5 server_cookie=72227e35a48f9948 in_seq=0 out_seq=0 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45b6a00640 10 client.6231 ms_handle_connect on v2:172.21.15.163:6833/1648150930 2024-07-26T06:17:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45b6a00640 1 -- 172.21.15.163:0/3930077096 <== mds.0 v2:172.21.15.163:6833/1648150930 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55df4c3256f0 con 0x55df4c420580 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45b6a00640 10 client.6231 handle_command_reply: tid=0 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45db7c5740 1 client.6231 shutdown 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 msgr2=0x55df4c4211c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.489+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c4211c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f45cc002230 tx=0x7f45cc04b680 comp rx=0 tx=0).stop 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 shutdown_connections 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:6833/1648150930,v1:172.21.15.163:6835/1648150930] conn(0x55df4c420580 0x55df4c420960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55df4c410aa0 0x55df4c4211c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4c3ff560 0x55df4c3e2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 --2- 172.21.15.163:0/3930077096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4c3b3990 0x55df4c47b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45b6a00640 0 client.6231 ms_handle_reset on v2:172.21.15.163:6833/1648150930 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 >> 172.21.15.163:0/3930077096 conn(0x55df4c3b9e50 msgr2=0x55df4c4100a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 shutdown_connections 2024-07-26T06:17:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 1 -- 172.21.15.163:0/3930077096 wait complete. 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.490+0000 7f45db7c5740 20 client.6231 trim_cache size 0 max 16384 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 6090, 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:17:53.496 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 6.6416340629999997, 2024-07-26T06:17:53.497 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 45, 2024-07-26T06:17:53.497 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 69, 2024-07-26T06:17:53.497 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 69, 2024-07-26T06:17:53.497 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 11.652151539 2024-07-26T06:17:53.497 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f45b0076630 msgr2=0x7f45b0078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f45b0076630 0x7f45b0078af0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f45d415d280 tx=0x7f45bc005c90 comp rx=0 tx=0).stop 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 msgr2=0x7f45d41477c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d41477c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c9a0 tx=0x7f45c400ce60 comp rx=0 tx=0).stop 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 shutdown_connections 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f45b0076630 0x7f45b0078af0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d41556f0 0x7f45d415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d40ce2d0 0x7f45d415c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 --2- 172.21.15.163:0/4055458052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d4071a70 0x7f45d41477c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 >> 172.21.15.163:0/4055458052 conn(0x7f45d40b0e30 msgr2=0x7f45d40b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 shutdown_connections 2024-07-26T06:17:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.491+0000 7f45d3200640 1 -- 172.21.15.163:0/4055458052 wait complete. 2024-07-26T06:17:53.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:17:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.705+0000 7f70b7e00640 1 -- 172.21.15.163:0/1836726128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 msgr2=0x7f70b8071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.705+0000 7f70b7e00640 1 --2- 172.21.15.163:0/1836726128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b8071530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f70b80cea70 tx=0x7f70a002a560 comp rx=0 tx=0).stop 2024-07-26T06:17:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 -- 172.21.15.163:0/1836726128 shutdown_connections 2024-07-26T06:17:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 --2- 172.21.15.163:0/1836726128 >> v2:172.21.15.164:3300/0 conn(0x7f70b8071a70 0x7f70b815ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 --2- 172.21.15.163:0/1836726128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b8071530 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 --2- 172.21.15.163:0/1836726128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70b81556f0 0x7f70b80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 -- 172.21.15.163:0/1836726128 >> 172.21.15.163:0/1836726128 conn(0x7f70b80b0e30 msgr2=0x7f70b80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 -- 172.21.15.163:0/1836726128 shutdown_connections 2024-07-26T06:17:53.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.706+0000 7f70b7e00640 1 -- 172.21.15.163:0/1836726128 wait complete. 2024-07-26T06:17:53.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.707+0000 7f70b7e00640 1 Processor -- start 2024-07-26T06:17:53.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.707+0000 7f70b7e00640 1 -- start start 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70b7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b8071a70 0x7f70b80ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70b7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70b7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70b81556f0 0x7f70b80ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70b7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70b815f2c0 con 0x7f70b80ce2d0 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70b7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f70b815f140 con 0x7f70b8071a70 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70b7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f70b815f440 con 0x7f70b81556f0 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.708+0000 7f70afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34300/0 (socket says 172.21.15.163:34300) 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 -- 172.21.15.163:0/2620499555 learned_addr learned my addr 172.21.15.163:0/2620499555 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:53.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 -- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70b81556f0 msgr2=0x7f70b80ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70b81556f0 0x7f70b80ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 -- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b8071a70 msgr2=0x7f70b80ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b8071a70 0x7f70b80ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 -- 172.21.15.163:0/2620499555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b814fb60 con 0x7f70b80ce2d0 2024-07-26T06:17:53.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.709+0000 7f70afe00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b814f620 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009850 tx=0x7f70a00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.710+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a003f070 con 0x7f70b80ce2d0 2024-07-26T06:17:53.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.710+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70a0004890 con 0x7f70b80ce2d0 2024-07-26T06:17:53.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.710+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0035030 con 0x7f70b80ce2d0 2024-07-26T06:17:53.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.710+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b80ab210 con 0x7f70b80ce2d0 2024-07-26T06:17:53.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.710+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b80ab7d0 con 0x7f70b80ce2d0 2024-07-26T06:17:53.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.711+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b815f960 con 0x7f70b80ce2d0 2024-07-26T06:17:53.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.712+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f70a0042070 con 0x7f70b80ce2d0 2024-07-26T06:17:53.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.713+0000 7f70aea00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7094076630 0x7f7094078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:53.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.713+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f70a00b8d60 con 0x7f70b80ce2d0 2024-07-26T06:17:53.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.713+0000 7f70b6a00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7094076630 0x7f7094078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:53.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.715+0000 7f70b6a00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7094076630 0x7f7094078af0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f70a800a860 tx=0x7f70a8005b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:53.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.717+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70a0082160 con 0x7f70b80ce2d0 2024-07-26T06:17:53.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.965+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f70b80b34f0 con 0x7f70b80ce2d0 2024-07-26T06:17:53.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.965+0000 7f70aea00640 1 -- 172.21.15.163:0/2620499555 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) ==== 53+0+73 (secure 0 0 0) 0x7f70a0081a60 con 0x7f70b80ce2d0 2024-07-26T06:17:53.966 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:17:53.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7094076630 msgr2=0x7f7094078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7094076630 0x7f7094078af0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f70a800a860 tx=0x7f70a8005b50 comp rx=0 tx=0).stop 2024-07-26T06:17:53.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 msgr2=0x7f70b814f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b814f620 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009850 tx=0x7f70a00039e0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 shutdown_connections 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f7094076630 0x7f7094078af0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70b81556f0 0x7f70b80ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70b80ce2d0 0x7f70b814f620 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 --2- 172.21.15.163:0/2620499555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70b8071a70 0x7f70b80ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.968+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 >> 172.21.15.163:0/2620499555 conn(0x7f70b80b0e30 msgr2=0x7f70b815b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.969+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 shutdown_connections 2024-07-26T06:17:53.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:53.969+0000 7f70b7e00640 1 -- 172.21.15.163:0/2620499555 wait complete. 2024-07-26T06:17:53.989 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:17:53.990 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:17:53.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:53.993 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.043 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42331sec preferred_lft 42331sec 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:17:54.044 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether c2:4f:6d:14:5d:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:17:54.045 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::c04f:6dff:fe14:5d96/64 scope link 2024-07-26T06:17:54.046 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:54.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:54.047 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:17:54.124 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:17:54.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:54.128 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:17:54.181 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:17:54.181 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:17:54.181 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:17:54.181 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:17:54.181 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:17:54.182 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:17:54.182 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:17:54.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:17:54.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:17:54.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:17:54.182 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:17:54.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:17:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:54.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:54.387 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:17:54.432 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:17:54.433 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:17:54.434 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:17:54.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:54.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:17:54.509 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:54.509 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:54.563 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:17:54.563 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:54.563 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:54.612 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:17:54.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:54.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:54.733 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:54.733 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:17:54.733 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:17:54.754 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:54.754 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:17:54.754 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:17:54.754 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:54.754 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:54.781 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:17:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:17:54.784 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:17:54.784 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:17:54.784 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:17:54.784 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:17:54.784 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:17:54.788 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:17:54.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:54.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:54.864 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:17:54.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:17:54.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:54.997 INFO:teuthology.orchestra.run.smithi163.stdout:client5287 2024-07-26T06:17:54.997 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:54.997 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:55.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:17:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 -- 172.21.15.163:0/972778066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 msgr2=0x7f2c700cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 --2- 172.21.15.163:0/972778066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c700cbab0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2c7006f590 tx=0x7f2c5c02a5f0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 -- 172.21.15.163:0/972778066 shutdown_connections 2024-07-26T06:17:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 --2- 172.21.15.163:0/972778066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c700cbab0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 --2- 172.21.15.163:0/972778066 >> v2:172.21.15.164:3300/0 conn(0x7f2c700741f0 0x7f2c700cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 -- 172.21.15.163:0/972778066 >> 172.21.15.163:0/972778066 conn(0x7f2c700b0e30 msgr2=0x7f2c700b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 -- 172.21.15.163:0/972778066 shutdown_connections 2024-07-26T06:17:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.299+0000 7f2c76c00640 1 -- 172.21.15.163:0/972778066 wait complete. 2024-07-26T06:17:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.301+0000 7f2c76c00640 1 Processor -- start 2024-07-26T06:17:55.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.301+0000 7f2c76c00640 1 -- start start 2024-07-26T06:17:55.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c700741f0 0x7f2c701e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:55.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c700cbff0 0x7f2c701e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:55.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c701e5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2c700b3c50 con 0x7f2c7015a6b0 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2c700b3ad0 con 0x7f2c700cbff0 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2c700b3dd0 con 0x7f2c700741f0 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c701e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c701e5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50522/0 (socket says 172.21.15.163:50522) 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c76200640 1 -- 172.21.15.163:0/1015718219 learned_addr learned my addr 172.21.15.163:0/1015718219 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c75800640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c700741f0 0x7f2c701e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.302+0000 7f2c74e00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c700cbff0 0x7f2c701e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76200640 1 -- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c700741f0 msgr2=0x7f2c701e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76200640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c700741f0 0x7f2c701e40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76200640 1 -- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c700cbff0 msgr2=0x7f2c701e45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76200640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c700cbff0 0x7f2c701e45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76200640 1 -- 172.21.15.163:0/1015718219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c701e5e70 con 0x7f2c7015a6b0 2024-07-26T06:17:55.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76200640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c701e5220 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c7a0 tx=0x7f2c6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:55.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68003b70 con 0x7f2c7015a6b0 2024-07-26T06:17:55.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c68003d30 con 0x7f2c7015a6b0 2024-07-26T06:17:55.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6800aab0 con 0x7f2c7015a6b0 2024-07-26T06:17:55.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c701e6150 con 0x7f2c7015a6b0 2024-07-26T06:17:55.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.303+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c701f3110 con 0x7f2c7015a6b0 2024-07-26T06:17:55.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.304+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c700cd0c0 con 0x7f2c7015a6b0 2024-07-26T06:17:55.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.306+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f2c6800ac50 con 0x7f2c7015a6b0 2024-07-26T06:17:55.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.307+0000 7f2c6d600640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c50076630 0x7f2c50078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:55.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.307+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f2c68097c20 con 0x7f2c7015a6b0 2024-07-26T06:17:55.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.307+0000 7f2c75800640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c50076630 0x7f2c50078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:55.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.308+0000 7f2c75800640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c50076630 0x7f2c50078af0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2c640058c0 tx=0x7f2c6400c440 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:55.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.311+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2c680610c0 con 0x7f2c7015a6b0 2024-07-26T06:17:55.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.559+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2c700b4aa0 con 0x7f2c7015a6b0 2024-07-26T06:17:55.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.560+0000 7f2c6d600640 1 -- 172.21.15.163:0/1015718219 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) ==== 53+0+73 (secure 0 0 0) 0x7f2c680690b0 con 0x7f2c7015a6b0 2024-07-26T06:17:55.561 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:17:55.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c50076630 msgr2=0x7f2c50078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:55.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c50076630 0x7f2c50078af0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2c640058c0 tx=0x7f2c6400c440 comp rx=0 tx=0).stop 2024-07-26T06:17:55.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 msgr2=0x7f2c701e5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:55.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c701e5220 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c7a0 tx=0x7f2c6800cc60 comp rx=0 tx=0).stop 2024-07-26T06:17:55.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 shutdown_connections 2024-07-26T06:17:55.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f2c50076630 0x7f2c50078af0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c7015a6b0 0x7f2c701e5220 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c700cbff0 0x7f2c701e45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 --2- 172.21.15.163:0/1015718219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c700741f0 0x7f2c701e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:55.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 >> 172.21.15.163:0/1015718219 conn(0x7f2c700b0e30 msgr2=0x7f2c70157e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:55.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 shutdown_connections 2024-07-26T06:17:55.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:55.563+0000 7f2c76c00640 1 -- 172.21.15.163:0/1015718219 wait complete. 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:17:55.584 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:17:55.585 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:17:55.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:55.585 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.643 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42307sec preferred_lft 42307sec 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:17:55.644 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:17:55.645 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::58b1:99ff:fee0:4877/64 scope link 2024-07-26T06:17:55.646 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:17:55.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:55.647 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:17:55.728 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:17:55.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:55.733 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:17:55.787 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:17:55.787 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:17:55.787 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:17:55.787 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:17:55.788 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:17:55.789 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:17:55.789 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:17:55.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:17:55.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:17:55.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:17:55.789 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:17:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:17:55.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:56.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:56.001 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:17:56.046 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:17:56.046 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:17:56.046 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:17:56.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:56.046 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:17:56.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:56.171 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:56.223 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:17:56.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:17:56.223 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:56.275 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:17:56.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:56.275 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:56.402 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:56.402 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:17:56.402 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:17:56.423 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:56.424 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:17:56.424 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:17:56.424 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:56.424 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:17:56.451 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:17:56.453 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:17:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:17:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:17:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:17:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:17:56.457 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:17:56.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:56.457 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:17:56.531 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:17:56.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:17:56.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:17:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:17:56.646 INFO:teuthology.orchestra.run.smithi164.stdout:client5293 2024-07-26T06:17:56.647 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:56.647 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:56.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-07-26T06:17:56.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:17:56.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 -- 172.21.15.163:0/3919062240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c070470 msgr2=0x7fa03c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 --2- 172.21.15.163:0/3919062240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c070470 0x7fa03c06bf40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa024009830 tx=0x7fa02402a510 comp rx=0 tx=0).stop 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 -- 172.21.15.163:0/3919062240 shutdown_connections 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 --2- 172.21.15.163:0/3919062240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c070470 0x7fa03c06bf40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 --2- 172.21.15.163:0/3919062240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa03c15b560 0x7fa03c072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 -- 172.21.15.163:0/3919062240 >> 172.21.15.163:0/3919062240 conn(0x7fa03c0b1f50 msgr2=0x7fa03c0b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 -- 172.21.15.163:0/3919062240 shutdown_connections 2024-07-26T06:17:56.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.916+0000 7fa042e00640 1 -- 172.21.15.163:0/3919062240 wait complete. 2024-07-26T06:17:56.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.918+0000 7fa042e00640 1 Processor -- start 2024-07-26T06:17:56.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.918+0000 7fa042e00640 1 -- start start 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa042e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03c070470 0x7fa03c145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa042e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 0x7fa03c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa042e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa03c15b560 0x7fa03c15a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa042e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c06c810 con 0x7fa03c074380 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa042e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c06c690 con 0x7fa03c070470 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa042e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa03c15f6d0 con 0x7fa03c15b560 2024-07-26T06:17:56.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa033e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 0x7fa03c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:56.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa033e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 0x7fa03c15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50540/0 (socket says 172.21.15.163:50540) 2024-07-26T06:17:56.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.920+0000 7fa033e00640 1 -- 172.21.15.163:0/3993544670 learned_addr learned my addr 172.21.15.163:0/3993544670 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:56.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa03be00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03c070470 0x7fa03c145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:56.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa040e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa03c15b560 0x7fa03c15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:56.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa033e00640 1 -- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa03c15b560 msgr2=0x7fa03c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:56.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa033e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa03c15b560 0x7fa03c15a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:56.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa033e00640 1 -- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03c070470 msgr2=0x7fa03c145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:56.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa033e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03c070470 0x7fa03c145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:56.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa033e00640 1 -- 172.21.15.163:0/3993544670 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03c15dcb0 con 0x7fa03c074380 2024-07-26T06:17:56.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.921+0000 7fa033e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 0x7fa03c15ed50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa024000c00 tx=0x7fa024032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:56.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.923+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024040070 con 0x7fa03c074380 2024-07-26T06:17:56.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.923+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa024032e70 con 0x7fa03c074380 2024-07-26T06:17:56.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.923+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024003d90 con 0x7fa03c074380 2024-07-26T06:17:56.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.923+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03c15df90 con 0x7fa03c074380 2024-07-26T06:17:56.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.924+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03c15e550 con 0x7fa03c074380 2024-07-26T06:17:56.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.925+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa03c06df10 con 0x7fa03c074380 2024-07-26T06:17:56.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.926+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fa024039040 con 0x7fa03c074380 2024-07-26T06:17:56.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.927+0000 7fa038c00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa018076660 0x7fa018078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:56.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.927+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fa0240b91d0 con 0x7fa03c074380 2024-07-26T06:17:56.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.927+0000 7fa03be00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa018076660 0x7fa018078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:56.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.929+0000 7fa03be00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa018076660 0x7fa018078b20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00bb50 tx=0x7fa02c00b2e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:56.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:56.931+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa03c06df10 con 0x7fa03c074380 2024-07-26T06:17:57.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.181+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fa03c0ae8f0 con 0x7fa03c074380 2024-07-26T06:17:57.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.933+0000 7fa038c00640 1 -- 172.21.15.163:0/3993544670 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v46) ==== 136+0+0 (secure 0 0 0) 0x7fa0240825d0 con 0x7fa03c074380 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.936+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa018076660 msgr2=0x7fa018078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.936+0000 7fa042e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa018076660 0x7fa018078b20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00bb50 tx=0x7fa02c00b2e0 comp rx=0 tx=0).stop 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.936+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 msgr2=0x7fa03c15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.936+0000 7fa042e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 0x7fa03c15ed50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa024000c00 tx=0x7fa024032c80 comp rx=0 tx=0).stop 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 shutdown_connections 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa018076660 0x7fa018078b20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa03c15b560 0x7fa03c15a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa03c074380 0x7fa03c15ed50 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 --2- 172.21.15.163:0/3993544670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03c070470 0x7fa03c145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 >> 172.21.15.163:0/3993544670 conn(0x7fa03c0b1f50 msgr2=0x7fa03c0afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 shutdown_connections 2024-07-26T06:17:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:57.937+0000 7fa042e00640 1 -- 172.21.15.163:0/3993544670 wait complete. 2024-07-26T06:17:57.961 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 -- 172.21.15.163:0/2142395676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01e00053c0 msgr2=0x7f01e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 --2- 172.21.15.163:0/2142395676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01e00053c0 0x7f01e000b9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f01f400c6f0 tx=0x7f01e0005830 comp rx=0 tx=0).stop 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 -- 172.21.15.163:0/2142395676 shutdown_connections 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 --2- 172.21.15.163:0/2142395676 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01e00053c0 0x7f01e000b9c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 -- 172.21.15.163:0/2142395676 >> 172.21.15.163:0/2142395676 conn(0x7f01fc0cfda0 msgr2=0x7f01fc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 -- 172.21.15.163:0/2142395676 shutdown_connections 2024-07-26T06:17:58.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.163+0000 7f01fb200640 1 -- 172.21.15.163:0/2142395676 wait complete. 2024-07-26T06:17:58.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.165+0000 7f01fb200640 1 Processor -- start 2024-07-26T06:17:58.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.165+0000 7f01fb200640 1 -- start start 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fb200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e00053c0 0x7f01fc159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fb200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1529c0 0x7f01fc1402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fb200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 0x7f01fc140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fb200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0ce680 con 0x7f01fc152390 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fb200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0ce500 con 0x7f01e00053c0 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fb200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01fc0ce800 con 0x7f01fc1529c0 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 0x7f01fc140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 0x7f01fc140800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50556/0 (socket says 172.21.15.163:50556) 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 -- 172.21.15.163:0/1121019480 learned_addr learned my addr 172.21.15.163:0/1121019480 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 -- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1529c0 msgr2=0x7f01fc1402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01f9400640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1529c0 0x7f01fc1402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01f9e00640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e00053c0 0x7f01fc159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1529c0 0x7f01fc1402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 -- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e00053c0 msgr2=0x7f01fc159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e00053c0 0x7f01fc159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.166+0000 7f01fa800640 1 -- 172.21.15.163:0/1121019480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc141200 con 0x7f01fc152390 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.167+0000 7f01fa800640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 0x7f01fc140800 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f01f4000c00 tx=0x7f01f40069a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.167+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f4003a00 con 0x7f01fc152390 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.167+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01f4003c90 con 0x7f01fc152390 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.167+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f4034670 con 0x7f01fc152390 2024-07-26T06:17:58.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.167+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc0cd060 con 0x7f01fc152390 2024-07-26T06:17:58.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.167+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc13e420 con 0x7f01fc152390 2024-07-26T06:17:58.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.168+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01fc0cd580 con 0x7f01fc152390 2024-07-26T06:17:58.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.169+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f01f4034810 con 0x7f01fc152390 2024-07-26T06:17:58.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.170+0000 7f01f2000640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f01d8076630 0x7f01d8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.170+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f01f40b79e0 con 0x7f01fc152390 2024-07-26T06:17:58.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.171+0000 7f01f9e00640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f01d8076630 0x7f01d8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.172+0000 7f01f9e00640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f01d8076630 0x7f01d8078af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006bd0 tx=0x7f01ec001400 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:58.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.174+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f01fc0cd580 con 0x7f01fc152390 2024-07-26T06:17:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.411+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f01fc13e660 con 0x7f01d8076630 2024-07-26T06:17:58.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.415+0000 7f01f2000640 1 -- 172.21.15.163:0/1121019480 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f01ec00e960 con 0x7f01d8076630 2024-07-26T06:17:58.416 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:17:58.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:17:58.416 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephfs" 2024-07-26T06:17:58.416 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:17:58.416 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f01d8076630 msgr2=0x7f01d8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f01d8076630 0x7f01d8078af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006bd0 tx=0x7f01ec001400 comp rx=0 tx=0).stop 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 msgr2=0x7f01fc140800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 0x7f01fc140800 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f01f4000c00 tx=0x7f01f40069a0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 shutdown_connections 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f01d8076630 0x7f01d8078af0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.417+0000 7f01fb200640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc152390 0x7f01fc140800 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.418+0000 7f01fb200640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1529c0 0x7f01fc1402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.418+0000 7f01fb200640 1 --2- 172.21.15.163:0/1121019480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e00053c0 0x7f01fc159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.418+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 >> 172.21.15.163:0/1121019480 conn(0x7f01fc0cfda0 msgr2=0x7f01fc15bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:58.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.418+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 shutdown_connections 2024-07-26T06:17:58.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.418+0000 7f01fb200640 1 -- 172.21.15.163:0/1121019480 wait complete. 2024-07-26T06:17:58.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:17:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 -- 172.21.15.163:0/3950709975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e60071a70 msgr2=0x7f6e6015ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 --2- 172.21.15.163:0/3950709975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e60071a70 0x7f6e6015ec00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6e6006ce80 tx=0x7f6e5802aa30 comp rx=0 tx=0).stop 2024-07-26T06:17:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 -- 172.21.15.163:0/3950709975 shutdown_connections 2024-07-26T06:17:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 --2- 172.21.15.163:0/3950709975 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e60071a70 0x7f6e6015ec00 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 --2- 172.21.15.163:0/3950709975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e600ce2d0 0x7f6e60071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 --2- 172.21.15.163:0/3950709975 >> v2:172.21.15.164:3300/0 conn(0x7f6e601556f0 0x7f6e600cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 -- 172.21.15.163:0/3950709975 >> 172.21.15.163:0/3950709975 conn(0x7f6e600b0e30 msgr2=0x7f6e600b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:58.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 -- 172.21.15.163:0/3950709975 shutdown_connections 2024-07-26T06:17:58.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.648+0000 7f6e5f200640 1 -- 172.21.15.163:0/3950709975 wait complete. 2024-07-26T06:17:58.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.649+0000 7f6e5f200640 1 Processor -- start 2024-07-26T06:17:58.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.650+0000 7f6e5f200640 1 -- start start 2024-07-26T06:17:58.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e60071a70 0x7f6e60147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e600ce2d0 0x7f6e600cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 0x7f6e6015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6015f2c0 con 0x7f6e601556f0 2024-07-26T06:17:58.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6015f140 con 0x7f6e600ce2d0 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6015f440 con 0x7f6e60071a70 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 0x7f6e6015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 0x7f6e6015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50572/0 (socket says 172.21.15.163:50572) 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 -- 172.21.15.163:0/2630510355 learned_addr learned my addr 172.21.15.163:0/2630510355 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5d400640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e600ce2d0 0x7f6e600cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5de00640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e60071a70 0x7f6e60147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 -- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e60071a70 msgr2=0x7f6e60147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e60071a70 0x7f6e60147d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 -- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e600ce2d0 msgr2=0x7f6e600cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e600ce2d0 0x7f6e600cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.651+0000 7f6e5e800640 1 -- 172.21.15.163:0/2630510355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e60148250 con 0x7f6e601556f0 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.652+0000 7f6e5e800640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 0x7f6e6015a050 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009d60 tx=0x7f6e58003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.652+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58047070 con 0x7f6e601556f0 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.652+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e58004250 con 0x7f6e601556f0 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.652+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58033050 con 0x7f6e601556f0 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.652+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e601f7800 con 0x7f6e601556f0 2024-07-26T06:17:58.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.652+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e601f7dc0 con 0x7f6e601556f0 2024-07-26T06:17:58.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.653+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e6015f960 con 0x7f6e601556f0 2024-07-26T06:17:58.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.654+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f6e5804a070 con 0x7f6e601556f0 2024-07-26T06:17:58.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.655+0000 7f6e56000640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6e3c076630 0x7f6e3c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:58.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.655+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f6e580b9080 con 0x7f6e601556f0 2024-07-26T06:17:58.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.656+0000 7f6e5de00640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6e3c076630 0x7f6e3c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:58.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.657+0000 7f6e5de00640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6e3c076630 0x7f6e3c078af0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6e480058c0 tx=0x7f6e48005850 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:58.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.660+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6e580824c0 con 0x7f6e601556f0 2024-07-26T06:17:58.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.867+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f6e600b34f0 con 0x7f6e601556f0 2024-07-26T06:17:58.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.867+0000 7f6e56000640 1 -- 172.21.15.163:0/2630510355 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f6e58081dc0 con 0x7f6e601556f0 2024-07-26T06:17:58.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6e3c076630 msgr2=0x7f6e3c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6e3c076630 0x7f6e3c078af0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6e480058c0 tx=0x7f6e48005850 comp rx=0 tx=0).stop 2024-07-26T06:17:58.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 msgr2=0x7f6e6015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:58.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 0x7f6e6015a050 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009d60 tx=0x7f6e58003f10 comp rx=0 tx=0).stop 2024-07-26T06:17:58.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 shutdown_connections 2024-07-26T06:17:58.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f6e3c076630 0x7f6e3c078af0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e601556f0 0x7f6e6015a050 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e600ce2d0 0x7f6e600cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 --2- 172.21.15.163:0/2630510355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e60071a70 0x7f6e60147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:58.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.870+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 >> 172.21.15.163:0/2630510355 conn(0x7f6e600b0e30 msgr2=0x7f6e6015d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:58.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.871+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 shutdown_connections 2024-07-26T06:17:58.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:58.871+0000 7f6e5f200640 1 -- 172.21.15.163:0/2630510355 wait complete. 2024-07-26T06:17:58.891 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:58.891 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:58.942 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:17:58.942 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:58.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:17:59.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 -- 172.21.15.163:0/639700956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4240ce4a0 msgr2=0x7fe424071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 --2- 172.21.15.163:0/639700956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4240ce4a0 0x7fe424071530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe410005fd0 tx=0x7fe41002a5d0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 -- 172.21.15.163:0/639700956 shutdown_connections 2024-07-26T06:17:59.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 --2- 172.21.15.163:0/639700956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe424071a70 0x7fe4240b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 --2- 172.21.15.163:0/639700956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4240ce4a0 0x7fe424071530 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 -- 172.21.15.163:0/639700956 >> 172.21.15.163:0/639700956 conn(0x7fe4240b0e30 msgr2=0x7fe4240b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:59.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 -- 172.21.15.163:0/639700956 shutdown_connections 2024-07-26T06:17:59.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.245+0000 7fe42a200640 1 -- 172.21.15.163:0/639700956 wait complete. 2024-07-26T06:17:59.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.247+0000 7fe42a200640 1 Processor -- start 2024-07-26T06:17:59.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.248+0000 7fe42a200640 1 -- start start 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe42a200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 0x7fe424150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe42a200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4240ce4a0 0x7fe4240aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe42a200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4241556f0 0x7fe4241e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe42a200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe4240b3fe0 con 0x7fe424071a70 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe42a200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe4240b3e60 con 0x7fe4241556f0 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe42a200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe4240b4160 con 0x7fe4240ce4a0 2024-07-26T06:17:59.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe428e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 0x7fe424150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe428e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 0x7fe424150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50586/0 (socket says 172.21.15.163:50586) 2024-07-26T06:17:59.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.249+0000 7fe428e00640 1 -- 172.21.15.163:0/1654359652 learned_addr learned my addr 172.21.15.163:0/1654359652 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe423e00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4240ce4a0 0x7fe4240aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe429800640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4241556f0 0x7fe4241e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe428e00640 1 -- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4240ce4a0 msgr2=0x7fe4240aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe428e00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4240ce4a0 0x7fe4240aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe428e00640 1 -- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4241556f0 msgr2=0x7fe4241e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe428e00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4241556f0 0x7fe4241e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe428e00640 1 -- 172.21.15.163:0/1654359652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe424150f50 con 0x7fe424071a70 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.250+0000 7fe428e00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 0x7fe424150a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe41800c430 tx=0x7fe41800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:59.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.251+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418017070 con 0x7fe424071a70 2024-07-26T06:17:59.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.251+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4180053e0 con 0x7fe424071a70 2024-07-26T06:17:59.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.251+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418003850 con 0x7fe424071a70 2024-07-26T06:17:59.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.251+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241e48f0 con 0x7fe424071a70 2024-07-26T06:17:59.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.251+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241e4eb0 con 0x7fe424071a70 2024-07-26T06:17:59.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.252+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4240b46f0 con 0x7fe424071a70 2024-07-26T06:17:59.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.253+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fe41801a070 con 0x7fe424071a70 2024-07-26T06:17:59.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.254+0000 7fe420c00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe404076630 0x7fe404078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.254+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fe4180970e0 con 0x7fe424071a70 2024-07-26T06:17:59.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.255+0000 7fe423e00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe404076630 0x7fe404078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.256+0000 7fe423e00640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe404076630 0x7fe404078af0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe410033000 tx=0x7fe410004730 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:59.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.261+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe4180605f0 con 0x7fe424071a70 2024-07-26T06:17:59.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.485+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe4240ced60 con 0x7fe424071a70 2024-07-26T06:17:59.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.486+0000 7fe420c00640 1 -- 172.21.15.163:0/1654359652 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) ==== 80+0+0 (secure 0 0 0) 0x7fe41805fef0 con 0x7fe424071a70 2024-07-26T06:17:59.487 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:17:59.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe404076630 msgr2=0x7fe404078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe404076630 0x7fe404078af0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe410033000 tx=0x7fe410004730 comp rx=0 tx=0).stop 2024-07-26T06:17:59.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 msgr2=0x7fe424150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 0x7fe424150a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe41800c430 tx=0x7fe41800c8f0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 shutdown_connections 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fe404076630 0x7fe404078af0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4241556f0 0x7fe4241e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4240ce4a0 0x7fe4240aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 --2- 172.21.15.163:0/1654359652 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe424071a70 0x7fe424150a10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.489+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 >> 172.21.15.163:0/1654359652 conn(0x7fe4240b0e30 msgr2=0x7fe4240cdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.490+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 shutdown_connections 2024-07-26T06:17:59.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.490+0000 7fe42a200640 1 -- 172.21.15.163:0/1654359652 wait complete. 2024-07-26T06:17:59.510 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:17:59.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:59.510 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-07-26T06:17:59.602 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:17:59.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:17:59.602 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:17:59.653 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:59.654 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:59.711 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:17:59.711 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:17:59.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 -- 172.21.15.163:0/21024832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 msgr2=0x7fb434004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 --2- 172.21.15.163:0/21024832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb434004c80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb43c0099a0 tx=0x7fb434005830 comp rx=0 tx=0).stop 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 -- 172.21.15.163:0/21024832 shutdown_connections 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 --2- 172.21.15.163:0/21024832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb434004c80 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 -- 172.21.15.163:0/21024832 >> 172.21.15.163:0/21024832 conn(0x7fb4500b0e30 msgr2=0x7fb4500b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 -- 172.21.15.163:0/21024832 shutdown_connections 2024-07-26T06:17:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.987+0000 7fb455400640 1 -- 172.21.15.163:0/21024832 wait complete. 2024-07-26T06:17:59.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.989+0000 7fb455400640 1 Processor -- start 2024-07-26T06:17:59.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.989+0000 7fb455400640 1 -- start start 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb455400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb4500c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb455400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb45013faa0 0x7fb45013f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb455400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4501453f0 0x7fb4500c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb455400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb4500cc0d0 con 0x7fb434004800 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb455400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb4500cbf50 con 0x7fb4501453f0 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb455400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb4500ad8e0 con 0x7fb45013faa0 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb4500c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb4500c4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50612/0 (socket says 172.21.15.163:50612) 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 -- 172.21.15.163:0/2580955646 learned_addr learned my addr 172.21.15.163:0/2580955646 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:17:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44f400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb45013faa0 0x7fb45013f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 -- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb45013faa0 msgr2=0x7fb45013f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb45013faa0 0x7fb45013f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb454a00640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4501453f0 0x7fb4500c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 -- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4501453f0 msgr2=0x7fb4500c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4501453f0 0x7fb4500c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.990+0000 7fb44fe00640 1 -- 172.21.15.163:0/2580955646 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4500c5090 con 0x7fb434004800 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb454a00640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4501453f0 0x7fb4500c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb44fe00640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb4500c4ab0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb44000c870 tx=0x7fb44000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440017070 con 0x7fb434004800 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb44000a640 con 0x7fb434004800 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44000aa50 con 0x7fb434004800 2024-07-26T06:17:59.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb450143240 con 0x7fb434004800 2024-07-26T06:17:59.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.991+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb450143800 con 0x7fb434004800 2024-07-26T06:17:59.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.992+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4500ae8b0 con 0x7fb434004800 2024-07-26T06:17:59.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.994+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7fb44000abf0 con 0x7fb434004800 2024-07-26T06:17:59.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.995+0000 7fb447e00640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb430076630 0x7fb430078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:17:59.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.995+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fb4400972e0 con 0x7fb434004800 2024-07-26T06:17:59.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.996+0000 7fb44f400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb430076630 0x7fb430078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:17:59.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.997+0000 7fb44f400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb430076630 0x7fb430078af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb43c034000 tx=0x7fb43c02c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:00.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:17:59.999+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb440060130 con 0x7fb434004800 2024-07-26T06:18:00.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.221+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb4500b2540 con 0x7fb434004800 2024-07-26T06:18:00.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.222+0000 7fb447e00640 1 -- 172.21.15.163:0/2580955646 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) ==== 80+0+0 (secure 0 0 0) 0x7fb440064430 con 0x7fb434004800 2024-07-26T06:18:00.223 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:18:00.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb430076630 msgr2=0x7fb430078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb430076630 0x7fb430078af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb43c034000 tx=0x7fb43c02c040 comp rx=0 tx=0).stop 2024-07-26T06:18:00.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 msgr2=0x7fb4500c4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb4500c4ab0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb44000c870 tx=0x7fb44000cd30 comp rx=0 tx=0).stop 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 shutdown_connections 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fb430076630 0x7fb430078af0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4501453f0 0x7fb4500c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb45013faa0 0x7fb45013f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 --2- 172.21.15.163:0/2580955646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb434004800 0x7fb4500c4ab0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.224+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 >> 172.21.15.163:0/2580955646 conn(0x7fb4500b0e30 msgr2=0x7fb4500b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.225+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 shutdown_connections 2024-07-26T06:18:00.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.225+0000 7fb455400640 1 -- 172.21.15.163:0/2580955646 wait complete. 2024-07-26T06:18:00.247 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:18:00.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:00.247 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-07-26T06:18:00.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:18:00.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:00.337 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:18:00.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:00.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.600+0000 7f44c7200640 1 -- 172.21.15.163:0/343380667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 msgr2=0x7f44a4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.600+0000 7f44c7200640 1 --2- 172.21.15.163:0/343380667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44a4003e70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f44b000c290 tx=0x7f44a4005830 comp rx=0 tx=0).stop 2024-07-26T06:18:00.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.600+0000 7f44c7200640 1 -- 172.21.15.163:0/343380667 shutdown_connections 2024-07-26T06:18:00.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.601+0000 7f44c7200640 1 --2- 172.21.15.163:0/343380667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44a4003e70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.601+0000 7f44c7200640 1 -- 172.21.15.163:0/343380667 >> 172.21.15.163:0/343380667 conn(0x7f44c006d420 msgr2=0x7f44c006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:00.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.601+0000 7f44c7200640 1 -- 172.21.15.163:0/343380667 shutdown_connections 2024-07-26T06:18:00.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.601+0000 7f44c7200640 1 -- 172.21.15.163:0/343380667 wait complete. 2024-07-26T06:18:00.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.602+0000 7f44c7200640 1 Processor -- start 2024-07-26T06:18:00.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.602+0000 7f44c7200640 1 -- start start 2024-07-26T06:18:00.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.603+0000 7f44c7200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44c015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:00.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.603+0000 7f44c7200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44c0156c50 0x7f44c0151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:00.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.603+0000 7f44c7200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44c0157f70 0x7f44c01583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:00.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.603+0000 7f44c7200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0159760 con 0x7f44a4003a50 2024-07-26T06:18:00.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.603+0000 7f44c7200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01595e0 con 0x7f44c0156c50 2024-07-26T06:18:00.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.603+0000 7f44c7200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f44c006c3b0 con 0x7f44c0157f70 2024-07-26T06:18:00.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44c015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:00.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44c015b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50640/0 (socket says 172.21.15.163:50640) 2024-07-26T06:18:00.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 -- 172.21.15.163:0/2078822578 learned_addr learned my addr 172.21.15.163:0/2078822578 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 -- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44c0157f70 msgr2=0x7f44c01583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44c0157f70 0x7f44c01583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 -- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44c0156c50 msgr2=0x7f44c0151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44c0156c50 0x7f44c0151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 -- 172.21.15.163:0/2078822578 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c015bad0 con 0x7f44a4003a50 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.604+0000 7f44bfe00640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44c015b3d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44b000c380 tx=0x7f44b0004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.605+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b0031040 con 0x7f44a4003a50 2024-07-26T06:18:00.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.605+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44b0006440 con 0x7f44a4003a50 2024-07-26T06:18:00.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.605+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b0001cb0 con 0x7f44a4003a50 2024-07-26T06:18:00.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.605+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c015bdb0 con 0x7f44a4003a50 2024-07-26T06:18:00.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.605+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c015c370 con 0x7f44a4003a50 2024-07-26T06:18:00.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.606+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44c00b40a0 con 0x7f44a4003a50 2024-07-26T06:18:00.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.607+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f44b0034070 con 0x7f44a4003a50 2024-07-26T06:18:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.608+0000 7f44b7e00640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f44a0076630 0x7f44a0078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.608+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f44b00b7c50 con 0x7f44a4003a50 2024-07-26T06:18:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.609+0000 7f44bf400640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f44a0076630 0x7f44a0078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:00.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.610+0000 7f44bf400640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f44a0076630 0x7f44a0078af0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f44c0158bc0 tx=0x7f44ac00b2f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.612+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44b0006260 con 0x7f44a4003a50 2024-07-26T06:18:00.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.855+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44c00b3b30 con 0x7f44a4003a50 2024-07-26T06:18:00.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.856+0000 7f44b7e00640 1 -- 172.21.15.163:0/2078822578 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7f44b00810e0 con 0x7f44a4003a50 2024-07-26T06:18:00.857 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:00.857 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:00.858 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:00.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f44a0076630 msgr2=0x7f44a0078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f44a0076630 0x7f44a0078af0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f44c0158bc0 tx=0x7f44ac00b2f0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 msgr2=0x7f44c015b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:00.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44c015b3d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44b000c380 tx=0x7f44b0004dd0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 shutdown_connections 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f44a0076630 0x7f44a0078af0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44c0157f70 0x7f44c01583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44c0156c50 0x7f44c0151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 --2- 172.21.15.163:0/2078822578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44a4003a50 0x7f44c015b3d0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 >> 172.21.15.163:0/2078822578 conn(0x7f44c006d420 msgr2=0x7f44c00ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 shutdown_connections 2024-07-26T06:18:00.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:00.859+0000 7f44c7200640 1 -- 172.21.15.163:0/2078822578 wait complete. 2024-07-26T06:18:00.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-07-26T06:18:01.105 INFO:teuthology.orchestra.run.smithi163.stdout:[] 2024-07-26T06:18:01.106 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-07-26T06:18:01.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol94510 2024-07-26T06:18:01.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 -- 172.21.15.163:0/2152622750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940cdbd0 msgr2=0x7f39940ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 --2- 172.21.15.163:0/2152622750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940cdbd0 0x7f39940ce080 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3990005fd0 tx=0x7f399002aa10 comp rx=0 tx=0).stop 2024-07-26T06:18:01.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 -- 172.21.15.163:0/2152622750 shutdown_connections 2024-07-26T06:18:01.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 --2- 172.21.15.163:0/2152622750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940cdbd0 0x7f39940ce080 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 --2- 172.21.15.163:0/2152622750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 0x7f399415c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 -- 172.21.15.163:0/2152622750 >> 172.21.15.163:0/2152622750 conn(0x7f399406f470 msgr2=0x7f399406f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:01.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 -- 172.21.15.163:0/2152622750 shutdown_connections 2024-07-26T06:18:01.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.318+0000 7f399b200640 1 -- 172.21.15.163:0/2152622750 wait complete. 2024-07-26T06:18:01.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.320+0000 7f399b200640 1 Processor -- start 2024-07-26T06:18:01.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.320+0000 7f399b200640 1 -- start start 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 0x7f399406e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39940cdbd0 0x7f399406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 0x7f39941f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f399415ca50 con 0x7f39940d0b00 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f399415c8d0 con 0x7f39940cdbd0 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f399415cbd0 con 0x7f3994073340 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 0x7f39941f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 0x7f39941f7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50660/0 (socket says 172.21.15.163:50660) 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.321+0000 7f399a800640 1 -- 172.21.15.163:0/816933316 learned_addr learned my addr 172.21.15.163:0/816933316 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:01.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f3999e00640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 0x7f399406e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f3999400640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39940cdbd0 0x7f399406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f399a800640 1 -- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 msgr2=0x7f399406e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f399a800640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 0x7f399406e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f399a800640 1 -- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39940cdbd0 msgr2=0x7f399406e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f399a800640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39940cdbd0 0x7f399406e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f399a800640 1 -- 172.21.15.163:0/816933316 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39940b4760 con 0x7f39940d0b00 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f3999e00640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 0x7f399406e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f3999400640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39940cdbd0 0x7f399406e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:01.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f399a800640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 0x7f39941f7390 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3990009d90 tx=0x7f3990003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:01.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.322+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990047070 con 0x7f39940d0b00 2024-07-26T06:18:01.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.323+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3990004250 con 0x7f39940d0b00 2024-07-26T06:18:01.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.323+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990033050 con 0x7f39940d0b00 2024-07-26T06:18:01.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.323+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39940b4a40 con 0x7f39940d0b00 2024-07-26T06:18:01.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.323+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39940b5000 con 0x7f39940d0b00 2024-07-26T06:18:01.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.324+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399415d180 con 0x7f39940d0b00 2024-07-26T06:18:01.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.325+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f399004a070 con 0x7f39940d0b00 2024-07-26T06:18:01.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.326+0000 7f398e000640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3978076660 0x7f3978078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.326+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f39900b8fb0 con 0x7f39940d0b00 2024-07-26T06:18:01.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.326+0000 7f3999e00640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3978076660 0x7f3978078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.328+0000 7f3999e00640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3978076660 0x7f3978078b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3988006fd0 tx=0x7f3988008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:01.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.330+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3990082440 con 0x7f39940d0b00 2024-07-26T06:18:01.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.566+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 --> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol94510", "target": ["mon-mgr", ""]}) -- 0x7f3994151770 con 0x7f3978076660 2024-07-26T06:18:01.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:18:01.582+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:01.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:18:01.582+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:01.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:18:01.582+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:01.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:18:01.582+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:01.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:18:01.582+0000 7f6cf2c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:01.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.669+0000 7f398e000640 1 -- 172.21.15.163:0/816933316 <== mgr.4115 v2:172.21.15.164:6816/1554374994 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3994151770 con 0x7f3978076660 2024-07-26T06:18:01.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.672+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3978076660 msgr2=0x7f3978078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.672+0000 7f399b200640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3978076660 0x7f3978078b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3988006fd0 tx=0x7f3988008040 comp rx=0 tx=0).stop 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 msgr2=0x7f39941f7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 0x7f39941f7390 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3990009d90 tx=0x7f3990003f10 comp rx=0 tx=0).stop 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 shutdown_connections 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f3978076660 0x7f3978078b20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39940d0b00 0x7f39941f7390 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39940cdbd0 0x7f399406e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 --2- 172.21.15.163:0/816933316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3994073340 0x7f399406e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 >> 172.21.15.163:0/816933316 conn(0x7f399406f470 msgr2=0x7f39940b26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 shutdown_connections 2024-07-26T06:18:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.673+0000 7f399b200640 1 -- 172.21.15.163:0/816933316 wait complete. 2024-07-26T06:18:01.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e8e00640 1 -- 172.21.15.163:0/3009986365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59c8003fd0 msgr2=0x7f59c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e8e00640 1 --2- 172.21.15.163:0/3009986365 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59c8003fd0 0x7f59c8004800 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00bbb0 tx=0x7f59c8005830 comp rx=0 tx=0).stop 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e3e00640 1 -- 172.21.15.163:0/3009986365 reap_dead start 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e8e00640 1 -- 172.21.15.163:0/3009986365 shutdown_connections 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e8e00640 1 -- 172.21.15.163:0/3009986365 >> 172.21.15.163:0/3009986365 conn(0x7f59e40cfda0 msgr2=0x7f59e40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e8e00640 1 -- 172.21.15.163:0/3009986365 shutdown_connections 2024-07-26T06:18:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.906+0000 7f59e8e00640 1 -- 172.21.15.163:0/3009986365 wait complete. 2024-07-26T06:18:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.908+0000 7f59e8e00640 1 Processor -- start 2024-07-26T06:18:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.908+0000 7f59e8e00640 1 -- start start 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e8e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59c8003fd0 0x7f59e40ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e8e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 0x7f59e415eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e8e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e415f3e0 0x7f59e40cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e8e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59e40ce470 con 0x7f59e415ea20 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e8e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f59e40ce2f0 con 0x7f59e415f3e0 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e8e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f59e40ce5f0 con 0x7f59c8003fd0 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 0x7f59e415eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 0x7f59e415eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50680/0 (socket says 172.21.15.163:50680) 2024-07-26T06:18:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 -- 172.21.15.163:0/1802264120 learned_addr learned my addr 172.21.15.163:0/1802264120 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59e3400640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59c8003fd0 0x7f59e40ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 -- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59c8003fd0 msgr2=0x7f59e40ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59c8003fd0 0x7f59e40ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 -- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e415f3e0 msgr2=0x7f59e40cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e415f3e0 0x7f59e40cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.909+0000 7f59daa00640 1 -- 172.21.15.163:0/1802264120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e415f860 con 0x7f59e415ea20 2024-07-26T06:18:01.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.910+0000 7f59daa00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 0x7f59e415eea0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00bb00 tx=0x7f59cc004a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:01.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.910+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc034040 con 0x7f59e415ea20 2024-07-26T06:18:01.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.910+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59cc03c070 con 0x7f59e415ea20 2024-07-26T06:18:01.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.910+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc0416d0 con 0x7f59e415ea20 2024-07-26T06:18:01.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.910+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e4146370 con 0x7f59e415ea20 2024-07-26T06:18:01.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.910+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e4158150 con 0x7f59e415ea20 2024-07-26T06:18:01.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.911+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e40b5090 con 0x7f59e415ea20 2024-07-26T06:18:01.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.912+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 11) ==== 98507+0+0 (secure 0 0 0) 0x7f59cc030420 con 0x7f59e415ea20 2024-07-26T06:18:01.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.913+0000 7f59dbe00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59c0076660 0x7f59c0078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:01.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.913+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f59cc0b8900 con 0x7f59e415ea20 2024-07-26T06:18:01.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.914+0000 7f59e3400640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59c0076660 0x7f59c0078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.915+0000 7f59e3400640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59c0076660 0x7f59c0078b20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f59e415cd80 tx=0x7f59d4005f90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.917+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f59e4146370 con 0x7f59e415ea20 2024-07-26T06:18:01.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:01.963+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 12) ==== 98558+0+0 (secure 0 0 0) 0x7f59cc07d9a0 con 0x7f59e415ea20 2024-07-26T06:18:02.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.169+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59e4157f30 con 0x7f59e415ea20 2024-07-26T06:18:02.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.170+0000 7f59dbe00640 1 -- 172.21.15.163:0/1802264120 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7f59cc081e10 con 0x7f59e415ea20 2024-07-26T06:18:02.170 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:02.171 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:02.171 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.172+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59c0076660 msgr2=0x7f59c0078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.172+0000 7f59e8e00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59c0076660 0x7f59c0078b20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f59e415cd80 tx=0x7f59d4005f90 comp rx=0 tx=0).stop 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.172+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 msgr2=0x7f59e415eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.172+0000 7f59e8e00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 0x7f59e415eea0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00bb00 tx=0x7f59cc004a30 comp rx=0 tx=0).stop 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 shutdown_connections 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f59c0076660 0x7f59c0078b20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e415f3e0 0x7f59e40cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59e415ea20 0x7f59e415eea0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 --2- 172.21.15.163:0/1802264120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59c8003fd0 0x7f59e40ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 >> 172.21.15.163:0/1802264120 conn(0x7f59e40cfda0 msgr2=0x7f59e40b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:02.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 shutdown_connections 2024-07-26T06:18:02.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.173+0000 7f59e8e00640 1 -- 172.21.15.163:0/1802264120 wait complete. 2024-07-26T06:18:02.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-07-26T06:18:02.409 INFO:teuthology.orchestra.run.smithi163.stdout:[{"id":5069,"entity":{"name":{"type":"client","num":5069},"addr":{"type":"any","addr":"172.21.15.164:0","nonce":1879372341}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.80798207799999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.960621526946994,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5069 172.21.15.164:0/1879372341","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"x","hostname":"smithi164","pid":"37768","root":"/"}}] 2024-07-26T06:18:02.410 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi164", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37768", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.164:0", "nonce": 1879372341, "type": "any" }, "name": { "num": 5069, "type": "client" } }, "id": 5069, "inst": "client.5069 172.21.15.164:0/1879372341", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.960621526946994 }, "state": "open", "uptime": 0.807982078 } ] 2024-07-26T06:18:02.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-07-26T06:18:02.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 -- 172.21.15.163:0/4151018458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa36815c4a0 msgr2=0x7fa3680cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:02.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 --2- 172.21.15.163:0/4151018458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa36815c4a0 0x7fa3680cc730 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa36815c8c0 tx=0x7fa35402aa10 comp rx=0 tx=0).stop 2024-07-26T06:18:02.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 -- 172.21.15.163:0/4151018458 shutdown_connections 2024-07-26T06:18:02.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 --2- 172.21.15.163:0/4151018458 >> v2:172.21.15.164:3300/0 conn(0x7fa3680ccc70 0x7fa36815ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 --2- 172.21.15.163:0/4151018458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa36815c4a0 0x7fa3680cc730 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 --2- 172.21.15.163:0/4151018458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa368140460 0x7fa36815bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 -- 172.21.15.163:0/4151018458 >> 172.21.15.163:0/4151018458 conn(0x7fa3680b1c40 msgr2=0x7fa3680b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:02.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 -- 172.21.15.163:0/4151018458 shutdown_connections 2024-07-26T06:18:02.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.610+0000 7fa36fe00640 1 -- 172.21.15.163:0/4151018458 wait complete. 2024-07-26T06:18:02.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.612+0000 7fa36fe00640 1 Processor -- start 2024-07-26T06:18:02.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.612+0000 7fa36fe00640 1 -- start start 2024-07-26T06:18:02.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3680ccc70 0x7fa3681f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:02.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 0x7fa368072550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:02.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa36815c4a0 0x7fa368072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:02.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa36815f500 con 0x7fa368140460 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa36815f380 con 0x7fa3680ccc70 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa36815f680 con 0x7fa36815c4a0 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 0x7fa368072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 0x7fa368072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50696/0 (socket says 172.21.15.163:50696) 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 -- 172.21.15.163:0/2850475609 learned_addr learned my addr 172.21.15.163:0/2850475609 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36d400640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3680ccc70 0x7fa3681f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 -- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa36815c4a0 msgr2=0x7fa368072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa36815c4a0 0x7fa368072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 -- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3680ccc70 msgr2=0x7fa3681f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3680ccc70 0x7fa3681f3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.614+0000 7fa36ca00640 1 -- 172.21.15.163:0/2850475609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3681f40e0 con 0x7fa368140460 2024-07-26T06:18:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.615+0000 7fa36ca00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 0x7fa368072550 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa354009bf0 tx=0x7fa354003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:02.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.615+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354033050 con 0x7fa368140460 2024-07-26T06:18:02.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.615+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa354004550 con 0x7fa368140460 2024-07-26T06:18:02.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.615+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354038450 con 0x7fa368140460 2024-07-26T06:18:02.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.615+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3680ae4c0 con 0x7fa368140460 2024-07-26T06:18:02.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.615+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3680aea80 con 0x7fa368140460 2024-07-26T06:18:02.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.617+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36815fbc0 con 0x7fa368140460 2024-07-26T06:18:02.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.618+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 12) ==== 98558+0+0 (secure 0 0 0) 0x7fa354036070 con 0x7fa368140460 2024-07-26T06:18:02.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.619+0000 7fa365600640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa3480766c0 0x7fa348078b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:02.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.619+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7fa3540b93d0 con 0x7fa368140460 2024-07-26T06:18:02.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.619+0000 7fa36d400640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa3480766c0 0x7fa348078b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:02.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.620+0000 7fa36d400640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa3480766c0 0x7fa348078b80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa3680af5d0 tx=0x7fa358008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:02.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.623+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa3681f40e0 con 0x7fa368140460 2024-07-26T06:18:02.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.886+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fa3680afad0 con 0x7fa368140460 2024-07-26T06:18:02.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.894+0000 7fa365600640 1 -- 172.21.15.163:0/2850475609 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v12) ==== 81+0+519625 (secure 0 0 0) 0x7fa354082690 con 0x7fa368140460 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 12, 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: "active_gid": 4115, 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: "active_name": "x", 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: "active_addrs": { 2024-07-26T06:18:02.900 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:6816", 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1554374994 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:6817", 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1554374994 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.901 INFO:teuthology.orchestra.run.smithi163.stdout: "active_addr": "172.21.15.164:6817/1554374994", 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "active_change": "2024-07-26T06:13:13.625578+0000", 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "active_mgr_features": 4540701547738038271, 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "available": true, 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "standbys": [ 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "gid": 4118, 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "y", 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_features": 4540701547738038271, 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "available_modules": [ 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "alerts", 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:02.902 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How frequently to reexamine health status", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.903 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:02.904 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.905 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.907 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.908 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_destination": { 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_destination", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Email address to send alerts to", 2024-07-26T06:18:02.909 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_from_name": { 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_from_name", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "Ceph", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.910 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Email From: name", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_host": { 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_host", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.911 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SMTP server", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_password": { 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_password", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.912 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Password to authenticate with", 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_port": { 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_port", 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "465", 2024-07-26T06:18:02.913 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SMTP port", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_sender": { 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_sender", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.914 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SMTP envelope sender", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_ssl": { 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_ssl", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.915 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use SSL to connect to SMTP server", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_user": { 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_user", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "User to authenticate as", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.917 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "balancer", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "active": { 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "active", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.919 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "automatically balance PGs across cluster", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "begin_time": { 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "begin_time", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0000", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.920 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "beginning time of day to automatically balance", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "begin_weekday": { 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "begin_weekday", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "0", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "6", 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.921 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_compat_max_iterations": { 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crush_compat_max_iterations", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "25", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "250", 2024-07-26T06:18:02.922 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_compat_metrics": { 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crush_compat_metrics", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "pgs,objects,bytes", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.923 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_compat_step": { 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crush_compat_step", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0.5", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "0.001", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "0.999", 2024-07-26T06:18:02.924 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "aggressiveness of optimization", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "end_time": { 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "end_time", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "2359", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.925 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "ending time of day to automatically balance", 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "end_weekday": { 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "end_weekday", 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.926 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "0", 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "6", 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.927 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:02.928 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.929 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.930 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.931 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.932 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "min_score": { 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "min_score", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.933 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "mode": { 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mode", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "upmap", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.934 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-compat", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "none", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "read", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap-read" 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Balancer mode", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_ids": { 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pool_ids", 2024-07-26T06:18:02.935 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:02.936 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:02.937 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap_max_deviation": { 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "upmap_max_deviation", 2024-07-26T06:18:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "deviation below which no optimization is attempted", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap_max_optimizations": { 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "upmap_max_optimizations", 2024-07-26T06:18:02.939 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephadm", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "agent_down_multiplier": { 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "agent_down_multiplier", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "3.0", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-07-26T06:18:02.941 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "agent_refresh_rate": { 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "agent_refresh_rate", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "20", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "agent_starting_port": { 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "agent_starting_port", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "4721", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.943 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_ptrace": { 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "allow_ptrace", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.944 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "autotune_interval": { 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "autotune_interval", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.945 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to autotune daemon memory", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "autotune_memory_target_ratio": { 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "autotune_memory_target_ratio", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0.7", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.946 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm_log_destination": { 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephadm_log_destination", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.947 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "file", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "file,syslog", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "syslog" 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "cgroups_split": { 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cgroups_split", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.948 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "config_checks_enabled": { 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "config_checks_enabled", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.949 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "config_dashboard": { 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "config_dashboard", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.950 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_alertmanager": { 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_alertmanager", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.951 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_base": { 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_base", 2024-07-26T06:18:02.952 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/ceph", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Container image name, without the tag", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_elasticsearch": { 2024-07-26T06:18:02.953 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_elasticsearch", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "elasticsearch container image", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_grafana": { 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_grafana", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.955 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_haproxy": { 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_haproxy", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "HAproxy container image", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.956 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_jaeger_agent": { 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_jaeger_agent", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Jaeger agent container image", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.957 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_jaeger_collector": { 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_jaeger_collector", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Jaeger collector container image", 2024-07-26T06:18:02.958 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_jaeger_query": { 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_jaeger_query", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Jaeger query container image", 2024-07-26T06:18:02.959 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_keepalived": { 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_keepalived", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Keepalived container image", 2024-07-26T06:18:02.960 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_loki": { 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_loki", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.961 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Loki container image", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_node_exporter": { 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_node_exporter", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.962 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_nvmeof": { 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_nvmeof", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Nvme-of container image", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_prometheus": { 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_prometheus", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.964 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_promtail": { 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_promtail", 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.965 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Promtail container image", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_samba": { 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_samba", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.966 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Samba/SMB container image", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_snmp_gateway": { 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_snmp_gateway", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.967 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SNMP Gateway container image", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "container_init": { 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_init", 2024-07-26T06:18:02.968 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Run podman/docker with `--init`", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "daemon_cache_timeout": { 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "daemon_cache_timeout", 2024-07-26T06:18:02.969 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "default_cephadm_command_timeout": { 2024-07-26T06:18:02.970 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "default_cephadm_command_timeout", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "900", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "default_registry": { 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "default_registry", 2024-07-26T06:18:02.971 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "device_cache_timeout": { 2024-07-26T06:18:02.972 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "device_cache_timeout", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "1800", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "seconds to cache device inventory", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "device_enhanced_scan": { 2024-07-26T06:18:02.973 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "device_enhanced_scan", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use libstoragemgmt during device scans", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.974 INFO:teuthology.orchestra.run.smithi163.stdout: "facts_cache_timeout": { 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "facts_cache_timeout", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "seconds to cache host facts data", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.975 INFO:teuthology.orchestra.run.smithi163.stdout: "grafana_dashboards_path": { 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "grafana_dashboards_path", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.976 INFO:teuthology.orchestra.run.smithi163.stdout: "host_check_interval": { 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "host_check_interval", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to perform a host check", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.977 INFO:teuthology.orchestra.run.smithi163.stdout: "hw_monitoring": { 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "hw_monitoring", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "inventory_list_all": { 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "inventory_list_all", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.979 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:02.980 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "log_refresh_metadata": { 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_refresh_metadata", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.981 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.982 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.983 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.984 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.985 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "manage_etc_ceph_ceph_conf": { 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.986 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "*", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.987 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "max_count_per_host": { 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_count_per_host", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "max number of daemons per service per host", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.988 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd_draining_count": { 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_osd_draining_count", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.989 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "migration_current": { 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "migration_current", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "internal - do not modify", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.990 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "mode": { 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mode", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "root", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm-package", 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: "root" 2024-07-26T06:18:02.991 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "mode for remote execution of cephadm", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "oob_default_addr": { 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "oob_default_addr", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "169.254.1.1", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.992 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Default address for RedFish API (oob management).", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "prometheus_alerts_path": { 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "prometheus_alerts_path", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.993 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_insecure": { 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_insecure", 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_password": { 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_password", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.995 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_url": { 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_url", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.996 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_username": { 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_username", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.997 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "secure_monitoring_stack": { 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "secure_monitoring_stack", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.998 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "service_discovery_port": { 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "service_discovery_port", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:02.999 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8765", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "cephadm service discovery port", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.000 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "ssh_config_file": { 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssh_config_file", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.001 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "use_agent": { 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "use_agent", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.002 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "use_repo_digest": { 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "use_repo_digest", 2024-07-26T06:18:03.003 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_on_failed_host_check": { 2024-07-26T06:18:03.004 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_on_failed_host_check", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise a health warning if the host check fails", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.005 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_on_stray_daemons": { 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_on_stray_daemons", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.006 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_on_stray_hosts": { 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_on_stray_hosts", 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.007 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.009 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.009 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-07-26T06:18:03.009 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crash", 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.010 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.011 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.012 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.013 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.014 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "retain_interval": { 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "retain_interval", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "31536000", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how long to retain crashes before pruning them", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.016 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_recent_interval": { 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_recent_interval", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "1209600", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "time interval in which to warn about recent crashes", 2024-07-26T06:18:03.018 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "dashboard", 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-07-26T06:18:03.019 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "ALERTMANAGER_API_HOST": { 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ALERTMANAGER_API_HOST", 2024-07-26T06:18:03.020 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-07-26T06:18:03.021 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.022 INFO:teuthology.orchestra.run.smithi163.stdout: "AUDIT_API_ENABLED": { 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "AUDIT_API_ENABLED", 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.023 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.024 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "ENABLE_BROWSABLE_API": { 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ENABLE_BROWSABLE_API", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.025 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-07-26T06:18:03.026 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.027 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.028 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.029 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.030 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_NFS": { 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.031 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_RBD": { 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_RGW": { 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_PASSWORD": { 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_PASSWORD", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "admin", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.036 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_URL": { 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_URL", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.037 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_USERNAME": { 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_USERNAME", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "admin", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.038 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.039 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.040 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.041 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "ISSUE_TRACKER_API_KEY": { 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.042 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "MANAGED_BY_CLUSTERS": { 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.043 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "[]", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "MULTICLUSTER_CONFIG": { 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "MULTICLUSTER_CONFIG", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.044 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "{}", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "PROMETHEUS_API_HOST": { 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PROMETHEUS_API_HOST", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.045 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.046 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.047 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.048 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-07-26T06:18:03.049 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-07-26T06:18:03.050 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.051 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.052 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_ENABLED": { 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_ENABLED", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.056 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.057 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.058 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "REST_REQUESTS_TIMEOUT": { 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "45", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.059 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_ACCESS_KEY": { 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_ACCESS_KEY", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.060 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "admin", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.061 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_SECRET_KEY": { 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_SECRET_KEY", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.062 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_SSL_VERIFY": { 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_SSL_VERIFY", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.063 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "UNSAFE_TLS_v1_2": { 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "UNSAFE_TLS_v1_2", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.064 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.065 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.066 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.067 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "cross_origin_url": { 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cross_origin_url", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.068 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "crt_file": { 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crt_file", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.069 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "debug": { 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "debug", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Enable/disable debug options", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "jwt_token_ttl": { 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "jwt_token_ttl", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.071 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "28800", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "key_file": { 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "key_file", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.072 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.073 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.074 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.076 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.077 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "motd": { 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "motd", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "The message of the day", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.079 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "redirect_resolve_ip_addr": { 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "redirect_resolve_ip_addr", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.080 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "server_addr": { 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_addr", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "::", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.081 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "server_port": { 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_port", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8080", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.082 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.083 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "ssl": { 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssl", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.084 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "ssl_server_port": { 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssl_server_port", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8443", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_behaviour": { 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_behaviour", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "redirect", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.086 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "redirect" 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_error_status_code": { 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_error_status_code", 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.087 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "500", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "400", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "599", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "url_prefix": { 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "url_prefix", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.088 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "devicehealth", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "enable_monitoring": { 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enable_monitoring", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.090 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "monitor device health metrics", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.091 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.092 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.094 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.095 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "mark_out_threshold": { 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mark_out_threshold", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.096 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "2419200", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": { 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pool_name", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.097 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "device_health_metrics", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "name of pool in which to store device health metrics", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "retention_period": { 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "retention_period", 2024-07-26T06:18:03.098 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "15552000", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how long to retain device health metrics", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "scrape_frequency": { 2024-07-26T06:18:03.099 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "scrape_frequency", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "86400", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to scrape device health metrics", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "self_heal": { 2024-07-26T06:18:03.100 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "self_heal", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.101 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to wake up and check device health", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.102 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.103 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_threshold": { 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_threshold", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "7257600", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.104 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "diskprediction_local", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.105 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.108 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.109 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "predict_interval": { 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "predict_interval", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "86400", 2024-07-26T06:18:03.110 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "predictor_model": { 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "predictor_model", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "prophetstor", 2024-07-26T06:18:03.111 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.112 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.113 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "influx", 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": false, 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "influxdb python module not found", 2024-07-26T06:18:03.114 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "batch_size": { 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "batch_size", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5000", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "database": { 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "database", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "ceph", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": { 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "hostname", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "InfluxDB server hostname", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.117 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "30", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "5", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.118 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.119 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.120 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.121 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.122 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.123 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "password": { 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "password", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "password of InfluxDB server user", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "port": { 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "port", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8086", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "InfluxDB server port", 2024-07-26T06:18:03.125 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.126 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "ssl": { 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssl", 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.127 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "false", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "threads": { 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "threads", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.128 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "32", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "username": { 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "username", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.129 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "username of InfluxDB server user", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "verify_ssl": { 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "verify_ssl", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.130 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "true", 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "insights", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.132 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.133 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.134 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.135 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.136 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.137 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.138 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "iostat", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.139 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.140 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.141 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.142 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.143 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.144 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "localpool", 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.145 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "failure_domain": { 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "failure_domain", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "host", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "failure domain for any created local pool", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-07-26T06:18:03.146 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.147 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.148 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.149 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.150 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.151 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": { 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "min_size", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.152 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "default min_size for any created local pool", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "num_rep": { 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "num_rep", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.153 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "3", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "default replica count for any created local pool", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": { 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pg_num", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.154 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "128", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "default pg_num for any created local pool", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "prefix": { 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "prefix", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.155 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "name prefix for any created local pool", 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.156 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "subtree": { 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "subtree", 2024-07-26T06:18:03.157 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "rack", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "CRUSH level for which to create a local pool", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mds_autoscaler", 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.159 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.161 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.163 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.164 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.165 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mirroring", 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.166 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.167 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.168 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.169 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.170 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.171 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.172 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "nfs", 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.173 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.174 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.175 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.176 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.177 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.178 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "orchestrator", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "fail_fs": { 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "fail_fs", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.180 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.181 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.182 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.184 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.185 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator": { 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "orchestrator", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.186 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "rook", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "test_orchestrator" 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Orchestrator backend", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "osd_perf_query", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.189 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.192 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.193 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.194 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "osd_support", 2024-07-26T06:18:03.195 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.196 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.197 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.198 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.199 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.200 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pg_autoscaler", 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.202 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.203 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.204 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.205 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.206 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "threshold": { 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "threshold", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "3.0", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1.0", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "scaling threshold", 2024-07-26T06:18:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "progress", 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_pg_recovery_event": { 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "allow_pg_recovery_event", 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "allow the module to show pg recovery progress", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "enabled": { 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enabled", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "max_completed_events": { 2024-07-26T06:18:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_completed_events", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "50", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "number of past completed events to remember", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how long the module is going to sleep", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "prometheus", 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "cache": { 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cache", 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.222 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "exclude_perf_counters": { 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "exclude_perf_counters", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.223 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.224 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.225 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.226 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.227 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_stats_pools": { 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_stats_pools", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_stats_pools_refresh_interval": { 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "300", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "scrape_interval": { 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "scrape_interval", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "15.0", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "server_addr": { 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_addr", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "::", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.234 INFO:teuthology.orchestra.run.smithi163.stdout: "server_port": { 2024-07-26T06:18:03.234 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_port", 2024-07-26T06:18:03.234 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.234 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.234 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "9283", 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.238 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.239 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "stale_cache_strategy": { 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "stale_cache_strategy", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "log", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.240 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_behaviour": { 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_behaviour", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "default", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "default", 2024-07-26T06:18:03.241 INFO:teuthology.orchestra.run.smithi163.stdout: "error" 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_error_status_code": { 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_error_status_code", 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "500", 2024-07-26T06:18:03.242 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "400", 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "599", 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_support", 2024-07-26T06:18:03.243 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.245 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.246 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.247 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.248 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "max_concurrent_snap_create": { 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_concurrent_snap_create", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "mirror_snapshot_schedule": { 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mirror_snapshot_schedule", 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.250 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.251 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "trash_purge_schedule": { 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "trash_purge_schedule", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.252 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "restful", 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "enable_auth": { 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enable_auth", 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.254 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "key_file": { 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "key_file", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.255 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.256 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.257 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.258 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.259 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.260 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "server_addr": { 2024-07-26T06:18:03.261 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_addr", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "server_port": { 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_port", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.264 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rgw", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.265 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.266 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.268 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.269 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.270 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rook", 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.271 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.272 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.273 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.274 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.275 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.276 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.277 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "storage_class": { 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "storage_class", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "local", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-07-26T06:18:03.278 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "selftest", 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.279 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.280 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.281 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.282 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.284 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "roption1": { 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "roption1", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.285 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "roption2": { 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "roption2", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "xyz", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.286 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption1": { 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption1", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.287 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption2": { 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption2", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.288 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption3": { 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption3", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.289 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption4": { 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption4", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption5": { 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption5", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.291 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption6": { 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption6", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.292 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption7": { 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption7", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "42", 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.293 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.294 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "testkey": { 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "testkey", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.295 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "testlkey": { 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "testlkey", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.296 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "testnewline": { 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "testnewline", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.297 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smb", 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.298 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.299 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.300 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.301 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.302 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.303 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.304 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "update_orchestration": { 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "update_orchestration", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.305 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "snap_schedule", 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_m_granularity": { 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "allow_m_granularity", 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "allow minute scheduled snapshots", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "dump_on_update": { 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "dump_on_update", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.307 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "dump database to debug log on update", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.308 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.309 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.310 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.311 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.313 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.314 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "stats", 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.315 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.316 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.317 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.318 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.319 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.320 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "status", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.322 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.324 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.325 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.326 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.327 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.328 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "telegraf", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "address": { 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "address", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.329 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "15", 2024-07-26T06:18:03.330 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.331 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.333 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.334 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.335 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.336 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.337 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "telemetry", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_basic": { 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_basic", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.338 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share basic cluster information (size, version)", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_crash": { 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_crash", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.339 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_device": { 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_device", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.340 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_ident": { 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_ident", 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.341 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_perf": { 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_perf", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.342 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share various performance metrics of a cluster", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "contact": { 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "contact", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.343 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "description": { 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "description", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.344 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "device_url": { 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "device_url", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.345 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "enabled": { 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enabled", 2024-07-26T06:18:03.346 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.347 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "24", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "8", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "last_opt_revision": { 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "last_opt_revision", 2024-07-26T06:18:03.348 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "1", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "leaderboard": { 2024-07-26T06:18:03.349 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "leaderboard", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.350 INFO:teuthology.orchestra.run.smithi163.stdout: "leaderboard_description": { 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "leaderboard_description", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.351 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.352 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.353 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.354 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.355 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.356 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "organization": { 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "organization", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.357 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "proxy": { 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "proxy", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.358 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.359 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "url": { 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "url", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-07-26T06:18:03.360 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "test_orchestrator", 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.361 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.362 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.363 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.364 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.365 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.366 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.367 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.368 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.369 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.371 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.372 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.373 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "max_concurrent_clones": { 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_concurrent_clones", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "4", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.374 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Number of asynchronous cloner threads", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "periodic_async_work": { 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "periodic_async_work", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.375 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Periodically check for async work", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot_clone_delay": { 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "snapshot_clone_delay", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.376 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot_clone_no_wait": { 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "snapshot_clone_no_wait", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.377 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.378 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.379 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "discovery_interval": { 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "discovery_interval", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "100", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.380 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "identifier": { 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "identifier", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.381 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.382 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.383 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.384 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.385 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.386 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "zabbix_host": { 2024-07-26T06:18:03.389 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix_host", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.390 INFO:teuthology.orchestra.run.smithi163.stdout: "zabbix_port": { 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix_port", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10051", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "zabbix_sender": { 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix_sender", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.392 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: "modules": [ 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: "iostat", 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: "nfs", 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: "restful" 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: "available_modules": [ 2024-07-26T06:18:03.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "alerts", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.394 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How frequently to reexamine health status", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.395 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.397 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.398 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.399 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_destination": { 2024-07-26T06:18:03.400 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_destination", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Email address to send alerts to", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.401 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_from_name": { 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_from_name", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "Ceph", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Email From: name", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_host": { 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_host", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SMTP server", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.403 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_password": { 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_password", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Password to authenticate with", 2024-07-26T06:18:03.404 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_port": { 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_port", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "465", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.405 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SMTP port", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_sender": { 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_sender", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.406 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SMTP envelope sender", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_ssl": { 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_ssl", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.407 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use SSL to connect to SMTP server", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "smtp_user": { 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smtp_user", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.408 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "User to authenticate as", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.409 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "balancer", 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "active": { 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "active", 2024-07-26T06:18:03.410 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "automatically balance PGs across cluster", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "begin_time": { 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "begin_time", 2024-07-26T06:18:03.411 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0000", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "beginning time of day to automatically balance", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "begin_weekday": { 2024-07-26T06:18:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "begin_weekday", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "0", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "6", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_compat_max_iterations": { 2024-07-26T06:18:03.413 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crush_compat_max_iterations", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "25", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "250", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.414 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_compat_metrics": { 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crush_compat_metrics", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "pgs,objects,bytes", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_compat_step": { 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crush_compat_step", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0.5", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "0.001", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "0.999", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "aggressiveness of optimization", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "end_time": { 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "end_time", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "2359", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "ending time of day to automatically balance", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "end_weekday": { 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "end_weekday", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "0", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "6", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-07-26T06:18:03.418 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.419 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.420 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.420 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.420 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.420 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.420 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.421 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.421 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.422 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.423 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.424 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "min_score": { 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "min_score", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.425 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "mode": { 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mode", 2024-07-26T06:18:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "upmap", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-compat", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "none", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "read", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap", 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap-read" 2024-07-26T06:18:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Balancer mode", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_ids": { 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pool_ids", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap_max_deviation": { 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "upmap_max_deviation", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "deviation below which no optimization is attempted", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "upmap_max_optimizations": { 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "upmap_max_optimizations", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephadm", 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "agent_down_multiplier": { 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "agent_down_multiplier", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "3.0", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "agent_refresh_rate": { 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "agent_refresh_rate", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "20", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-07-26T06:18:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "agent_starting_port": { 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "agent_starting_port", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "4721", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_ptrace": { 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "allow_ptrace", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "autotune_interval": { 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "autotune_interval", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to autotune daemon memory", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "autotune_memory_target_ratio": { 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "autotune_memory_target_ratio", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0.7", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm_log_destination": { 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephadm_log_destination", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "file", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "file,syslog", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "syslog" 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "cgroups_split": { 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cgroups_split", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "config_checks_enabled": { 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "config_checks_enabled", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "config_dashboard": { 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "config_dashboard", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_alertmanager": { 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_alertmanager", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_base": { 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_base", 2024-07-26T06:18:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/ceph", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Container image name, without the tag", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_elasticsearch": { 2024-07-26T06:18:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_elasticsearch", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "elasticsearch container image", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_grafana": { 2024-07-26T06:18:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_grafana", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_haproxy": { 2024-07-26T06:18:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_haproxy", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "HAproxy container image", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_jaeger_agent": { 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_jaeger_agent", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Jaeger agent container image", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_jaeger_collector": { 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_jaeger_collector", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Jaeger collector container image", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_jaeger_query": { 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_jaeger_query", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Jaeger query container image", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_keepalived": { 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_keepalived", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Keepalived container image", 2024-07-26T06:18:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_loki": { 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_loki", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Loki container image", 2024-07-26T06:18:03.454 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_node_exporter": { 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_node_exporter", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_nvmeof": { 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_nvmeof", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Nvme-of container image", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_prometheus": { 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_prometheus", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Prometheus container image", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_promtail": { 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_promtail", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Promtail container image", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_samba": { 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_samba", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Samba/SMB container image", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image_snmp_gateway": { 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_image_snmp_gateway", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "SNMP Gateway container image", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "container_init": { 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "container_init", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Run podman/docker with `--init`", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "daemon_cache_timeout": { 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "daemon_cache_timeout", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "default_cephadm_command_timeout": { 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "default_cephadm_command_timeout", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "900", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "default_registry": { 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "default_registry", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "docker.io", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "device_cache_timeout": { 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "device_cache_timeout", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "1800", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "seconds to cache device inventory", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "device_enhanced_scan": { 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "device_enhanced_scan", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use libstoragemgmt during device scans", 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "facts_cache_timeout": { 2024-07-26T06:18:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "facts_cache_timeout", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "seconds to cache host facts data", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "grafana_dashboards_path": { 2024-07-26T06:18:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "grafana_dashboards_path", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "host_check_interval": { 2024-07-26T06:18:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "host_check_interval", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to perform a host check", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "hw_monitoring": { 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "hw_monitoring", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "inventory_list_all": { 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "inventory_list_all", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "log_refresh_metadata": { 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_refresh_metadata", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "manage_etc_ceph_ceph_conf": { 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "*", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "max_count_per_host": { 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_count_per_host", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "max number of daemons per service per host", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd_draining_count": { 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_osd_draining_count", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "migration_current": { 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "migration_current", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "internal - do not modify", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "mode": { 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mode", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "root", 2024-07-26T06:18:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm-package", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "root" 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "mode for remote execution of cephadm", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "oob_default_addr": { 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "oob_default_addr", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "169.254.1.1", 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Default address for RedFish API (oob management).", 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "prometheus_alerts_path": { 2024-07-26T06:18:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "prometheus_alerts_path", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_insecure": { 2024-07-26T06:18:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_insecure", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_password": { 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_password", 2024-07-26T06:18:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_url": { 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_url", 2024-07-26T06:18:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "registry_username": { 2024-07-26T06:18:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "registry_username", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "secure_monitoring_stack": { 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "secure_monitoring_stack", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "service_discovery_port": { 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "service_discovery_port", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8765", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "cephadm service discovery port", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "ssh_config_file": { 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssh_config_file", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "use_agent": { 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "use_agent", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "use_repo_digest": { 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "use_repo_digest", 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_on_failed_host_check": { 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_on_failed_host_check", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise a health warning if the host check fails", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_on_stray_daemons": { 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_on_stray_daemons", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_on_stray_hosts": { 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_on_stray_hosts", 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crash", 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "retain_interval": { 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "retain_interval", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "31536000", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how long to retain crashes before pruning them", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_recent_interval": { 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_recent_interval", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "1209600", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "time interval in which to warn about recent crashes", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "dashboard", 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "ALERTMANAGER_API_HOST": { 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ALERTMANAGER_API_HOST", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-07-26T06:18:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "AUDIT_API_ENABLED": { 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "AUDIT_API_ENABLED", 2024-07-26T06:18:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-07-26T06:18:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "ENABLE_BROWSABLE_API": { 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ENABLE_BROWSABLE_API", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_NFS": { 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_RBD": { 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "FEATURE_TOGGLE_RGW": { 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_PASSWORD": { 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_PASSWORD", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "admin", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_URL": { 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_URL", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_API_USERNAME": { 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_API_USERNAME", 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "admin", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-07-26T06:18:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "ISSUE_TRACKER_API_KEY": { 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-07-26T06:18:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "MANAGED_BY_CLUSTERS": { 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-07-26T06:18:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "[]", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "MULTICLUSTER_CONFIG": { 2024-07-26T06:18:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "MULTICLUSTER_CONFIG", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "{}", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "PROMETHEUS_API_HOST": { 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PROMETHEUS_API_HOST", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.543 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.544 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_ENABLED": { 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_ENABLED", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.545 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.546 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.547 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8", 2024-07-26T06:18:03.548 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "REST_REQUESTS_TIMEOUT": { 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.549 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "45", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_ACCESS_KEY": { 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_ACCESS_KEY", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.550 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.551 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "admin", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_SECRET_KEY": { 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_SECRET_KEY", 2024-07-26T06:18:03.552 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "RGW_API_SSL_VERIFY": { 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "RGW_API_SSL_VERIFY", 2024-07-26T06:18:03.553 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "UNSAFE_TLS_v1_2": { 2024-07-26T06:18:03.554 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "UNSAFE_TLS_v1_2", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.555 INFO:teuthology.orchestra.run.smithi163.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "cross_origin_url": { 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cross_origin_url", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "crt_file": { 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "crt_file", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.560 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "debug": { 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "debug", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Enable/disable debug options", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.561 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "jwt_token_ttl": { 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "jwt_token_ttl", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "28800", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.562 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "key_file": { 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "key_file", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.563 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.564 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.566 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.567 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.568 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "motd": { 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "motd", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.569 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "The message of the day", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "redirect_resolve_ip_addr": { 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "redirect_resolve_ip_addr", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.570 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "server_addr": { 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_addr", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.571 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "::", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "server_port": { 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_port", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.572 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8080", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.573 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "ssl": { 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssl", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.574 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "ssl_server_port": { 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssl_server_port", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.575 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8443", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_behaviour": { 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_behaviour", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.576 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "redirect", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "redirect" 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.577 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_error_status_code": { 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_error_status_code", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "500", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "400", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "599", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "url_prefix": { 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "url_prefix", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.579 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "devicehealth", 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "enable_monitoring": { 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enable_monitoring", 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.580 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "monitor device health metrics", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.581 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.582 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.583 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.584 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.585 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.586 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "mark_out_threshold": { 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mark_out_threshold", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "2419200", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-07-26T06:18:03.587 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": { 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pool_name", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "device_health_metrics", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "name of pool in which to store device health metrics", 2024-07-26T06:18:03.588 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "retention_period": { 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "retention_period", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "15552000", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.589 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how long to retain device health metrics", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "scrape_frequency": { 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "scrape_frequency", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "86400", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.590 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to scrape device health metrics", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "self_heal": { 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "self_heal", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.591 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.592 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how frequently to wake up and check device health", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.593 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "warn_threshold": { 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "warn_threshold", 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.594 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "7257600", 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "diskprediction_local", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.596 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.597 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.599 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.600 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "predict_interval": { 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "predict_interval", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.601 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "86400", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "predictor_model": { 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "predictor_model", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.602 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "prophetstor", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.603 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "600", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.604 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "influx", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": false, 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "influxdb python module not found", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "batch_size": { 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "batch_size", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5000", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.606 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "database": { 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "database", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "ceph", 2024-07-26T06:18:03.607 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": { 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "hostname", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.608 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "InfluxDB server hostname", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "30", 2024-07-26T06:18:03.609 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "5", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.610 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.611 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.612 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.613 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.614 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "password": { 2024-07-26T06:18:03.615 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "password", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "password of InfluxDB server user", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "port": { 2024-07-26T06:18:03.616 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "port", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "8086", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "InfluxDB server port", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "ssl": { 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "ssl", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "false", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "threads": { 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "threads", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "32", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.620 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "username": { 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "username", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "username of InfluxDB server user", 2024-07-26T06:18:03.621 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "verify_ssl": { 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "verify_ssl", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "true", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.622 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "insights", 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.623 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.624 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.624 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.624 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.624 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.624 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.624 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.625 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.625 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.626 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.627 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.628 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.629 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.630 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "iostat", 2024-07-26T06:18:03.631 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.633 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.635 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.636 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.637 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "localpool", 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.638 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "failure_domain": { 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "failure_domain", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "host", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "failure domain for any created local pool", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.639 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.641 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.642 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.643 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.644 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": { 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "min_size", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.645 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "default min_size for any created local pool", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "num_rep": { 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "num_rep", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "3", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.646 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "default replica count for any created local pool", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": { 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pg_num", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "128", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.647 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "default pg_num for any created local pool", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "prefix": { 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "prefix", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.648 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "name prefix for any created local pool", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.649 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "subtree": { 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "subtree", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "rack", 2024-07-26T06:18:03.650 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "CRUSH level for which to create a local pool", 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mds_autoscaler", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.652 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.654 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.655 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.656 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.657 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mirroring", 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.658 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.659 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.660 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.661 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.662 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.663 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "nfs", 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.665 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.666 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.667 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.668 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.669 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.670 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.671 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "orchestrator", 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "fail_fs": { 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "fail_fs", 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.672 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.673 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.674 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.675 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.676 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.677 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator": { 2024-07-26T06:18:03.678 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "orchestrator", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "rook", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "test_orchestrator" 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Orchestrator backend", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.680 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "osd_perf_query", 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.681 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.682 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.683 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.684 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.685 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.686 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "osd_support", 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.688 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.689 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.690 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.691 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.692 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.693 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.694 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "pg_autoscaler", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.695 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.696 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.697 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.698 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.699 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.700 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.701 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "threshold": { 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "threshold", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "3.0", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1.0", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.702 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "scaling threshold", 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "progress", 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_pg_recovery_event": { 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "allow_pg_recovery_event", 2024-07-26T06:18:03.703 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "allow the module to show pg recovery progress", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "enabled": { 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enabled", 2024-07-26T06:18:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.705 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.706 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.707 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.708 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.709 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "max_completed_events": { 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_completed_events", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "50", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "number of past completed events to remember", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.711 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "sleep_interval": { 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sleep_interval", 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "5", 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.712 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "how long the module is going to sleep", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.713 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "prometheus", 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.714 INFO:teuthology.orchestra.run.smithi163.stdout: "cache": { 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cache", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "exclude_perf_counters": { 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "exclude_perf_counters", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.716 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.717 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.718 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.719 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.720 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.721 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_stats_pools": { 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_stats_pools", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_stats_pools_refresh_interval": { 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "300", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "scrape_interval": { 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "scrape_interval", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "15.0", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.724 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "server_addr": { 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_addr", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "::", 2024-07-26T06:18:03.725 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "server_port": { 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_port", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "9283", 2024-07-26T06:18:03.726 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.727 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "stale_cache_strategy": { 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "stale_cache_strategy", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "log", 2024-07-26T06:18:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_behaviour": { 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_behaviour", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "default", 2024-07-26T06:18:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "default", 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "error" 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "standby_error_status_code": { 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "standby_error_status_code", 2024-07-26T06:18:03.730 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "500", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "400", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "599", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.731 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_support", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.732 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.733 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.734 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.735 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.736 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "max_concurrent_snap_create": { 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_concurrent_snap_create", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.737 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "mirror_snapshot_schedule": { 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mirror_snapshot_schedule", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.738 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.739 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "trash_purge_schedule": { 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "trash_purge_schedule", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.740 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "restful", 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "enable_auth": { 2024-07-26T06:18:03.741 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enable_auth", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "key_file": { 2024-07-26T06:18:03.742 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "key_file", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.743 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.744 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.745 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.746 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.747 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.748 INFO:teuthology.orchestra.run.smithi163.stdout: "server_addr": { 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_addr", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "server_port": { 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "server_port", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.750 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.751 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rgw", 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.752 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.753 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.754 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.755 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.757 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rook", 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.759 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.760 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.761 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.762 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.763 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.764 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "storage_class": { 2024-07-26T06:18:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "storage_class", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "local", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "selftest", 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.767 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.769 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.770 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.771 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "roption1": { 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "roption1", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.772 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "roption2": { 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "roption2", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.773 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "xyz", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption1": { 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption1", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.774 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption2": { 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption2", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.775 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption3": { 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption3", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "float", 2024-07-26T06:18:03.776 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption4": { 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption4", 2024-07-26T06:18:03.777 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.778 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption5": { 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption5", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.779 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption6": { 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption6", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.780 INFO:teuthology.orchestra.run.smithi163.stdout: "rwoption7": { 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rwoption7", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "1", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "42", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.781 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.782 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "testkey": { 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "testkey", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.783 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "testlkey": { 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "testlkey", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.784 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "testnewline": { 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "testnewline", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.785 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "smb", 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.786 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.787 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.788 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.789 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "update_orchestration": { 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "update_orchestration", 2024-07-26T06:18:03.792 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "snap_schedule", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "allow_m_granularity": { 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "allow_m_granularity", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "allow minute scheduled snapshots", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "dump_on_update": { 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "dump_on_update", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.795 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "dump database to debug log on update", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.796 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.797 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.800 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.801 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "stats", 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.802 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.803 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.804 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.805 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.806 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.807 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.808 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "status", 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.809 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.810 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.811 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.812 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.813 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.814 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "telegraf", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "address": { 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "address", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.816 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "15", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.817 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.818 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.821 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.822 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.823 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.824 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.824 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.824 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.824 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.825 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.825 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "telemetry", 2024-07-26T06:18:03.826 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_basic": { 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_basic", 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.827 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share basic cluster information (size, version)", 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.828 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_crash": { 2024-07-26T06:18:03.829 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_crash", 2024-07-26T06:18:03.829 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.829 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.829 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.829 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.829 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.831 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_device": { 2024-07-26T06:18:03.831 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_device", 2024-07-26T06:18:03.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.831 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.831 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.832 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.832 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.832 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.905+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa3480766c0 msgr2=0x7fa348078b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:03.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.905+0000 7fa36fe00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa3480766c0 0x7fa348078b80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa3680af5d0 tx=0x7fa358008040 comp rx=0 tx=0).stop 2024-07-26T06:18:03.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.905+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 msgr2=0x7fa368072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:03.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.905+0000 7fa36fe00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 0x7fa368072550 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa354009bf0 tx=0x7fa354003f90 comp rx=0 tx=0).stop 2024-07-26T06:18:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 shutdown_connections 2024-07-26T06:18:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7fa3480766c0 0x7fa348078b80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa36815c4a0 0x7fa368072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa368140460 0x7fa368072550 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 --2- 172.21.15.163:0/2850475609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3680ccc70 0x7fa3681f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 >> 172.21.15.163:0/2850475609 conn(0x7fa3680b1c40 msgr2=0x7fa3680b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:03.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 shutdown_connections 2024-07-26T06:18:03.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:02.906+0000 7fa36fe00640 1 -- 172.21.15.163:0/2850475609 wait complete. 2024-07-26T06:18:03.836 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-07-26T06:18:03.836 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.836 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.837 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.837 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_ident": { 2024-07-26T06:18:03.837 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_ident", 2024-07-26T06:18:03.837 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-07-26T06:18:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.840 INFO:teuthology.orchestra.run.smithi163.stdout: "channel_perf": { 2024-07-26T06:18:03.840 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "channel_perf", 2024-07-26T06:18:03.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.840 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.840 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.840 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.841 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.841 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.841 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.841 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Share various performance metrics of a cluster", 2024-07-26T06:18:03.841 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.841 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.842 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.842 INFO:teuthology.orchestra.run.smithi163.stdout: "contact": { 2024-07-26T06:18:03.842 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "contact", 2024-07-26T06:18:03.842 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.842 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.843 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.843 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.843 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.843 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.843 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: "description": { 2024-07-26T06:18:03.844 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "description", 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.845 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: "device_url": { 2024-07-26T06:18:03.846 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "device_url", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "enabled": { 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "enabled", 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.848 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "24", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "8", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.849 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "last_opt_revision": { 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "last_opt_revision", 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "1", 2024-07-26T06:18:03.850 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "leaderboard": { 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "leaderboard", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.851 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.853 INFO:teuthology.orchestra.run.smithi163.stdout: "leaderboard_description": { 2024-07-26T06:18:03.853 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "leaderboard_description", 2024-07-26T06:18:03.853 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.853 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.853 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.853 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.854 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.854 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.854 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.854 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.854 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.854 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.855 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.856 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.857 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.858 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.859 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.860 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.861 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.861 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.861 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.861 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.861 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.862 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.862 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.862 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.862 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.862 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.862 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "organization": { 2024-07-26T06:18:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "organization", 2024-07-26T06:18:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.864 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.865 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.865 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.865 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.865 INFO:teuthology.orchestra.run.smithi163.stdout: "proxy": { 2024-07-26T06:18:03.866 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "proxy", 2024-07-26T06:18:03.866 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.866 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.866 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.866 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.866 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.867 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.867 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.867 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.867 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.867 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.868 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.869 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.869 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.869 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.869 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.869 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.869 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: "url": { 2024-07-26T06:18:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "url", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "test_orchestrator", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.872 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.873 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.874 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.875 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.876 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.877 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.878 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.879 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.880 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.881 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "max_concurrent_clones": { 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "max_concurrent_clones", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "4", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Number of asynchronous cloner threads", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "periodic_async_work": { 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "periodic_async_work", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Periodically check for async work", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot_clone_delay": { 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "snapshot_clone_delay", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.886 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot_clone_no_wait": { 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "snapshot_clone_no_wait", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "True", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.887 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix", 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "can_run": true, 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "error_string": "", 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "module_options": { 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "discovery_interval": { 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "discovery_interval", 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "uint", 2024-07-26T06:18:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "100", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "identifier": { 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "identifier", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.890 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "interval": { 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "interval", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "secs", 2024-07-26T06:18:03.891 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "60", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "log_level": { 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_level", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.892 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.893 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster": { 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.894 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_cluster_level": { 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_cluster_level", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "info", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [ 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "", 2024-07-26T06:18:03.895 INFO:teuthology.orchestra.run.smithi163.stdout: "critical", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "debug", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "error", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "info", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "warning" 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": { 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "log_to_file", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "bool", 2024-07-26T06:18:03.896 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "False", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "sqlite3_killpoint": { 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "sqlite3_killpoint", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.897 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "dev", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "0", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "zabbix_host": { 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix_host", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.898 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "zabbix_port": { 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix_port", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "int", 2024-07-26T06:18:03.899 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "10051", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "zabbix_sender": { 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "zabbix_sender", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "str", 2024-07-26T06:18:03.900 INFO:teuthology.orchestra.run.smithi163.stdout: "level": "advanced", 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 0, 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "min": "", 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "max": "", 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "enum_allowed": [], 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "desc": "", 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "long_desc": "", 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "tags": [], 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "see_also": [] 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.901 INFO:teuthology.orchestra.run.smithi163.stdout: "services": {}, 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "always_on_modules": { 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "octopus": [ 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "balancer", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "crash", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "devicehealth", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscaler", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "progress", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_support", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "status", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "telemetry", 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "volumes" 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.902 INFO:teuthology.orchestra.run.smithi163.stdout: "pacific": [ 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "balancer", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "crash", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "devicehealth", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscaler", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "progress", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_support", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "status", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "telemetry", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "volumes" 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "quincy": [ 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "balancer", 2024-07-26T06:18:03.903 INFO:teuthology.orchestra.run.smithi163.stdout: "crash", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "devicehealth", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscaler", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "progress", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_support", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "status", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "telemetry", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "volumes" 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "reef": [ 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "balancer", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "crash", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "devicehealth", 2024-07-26T06:18:03.904 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscaler", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "progress", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_support", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "status", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "telemetry", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "volumes" 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "squid": [ 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "balancer", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "crash", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "devicehealth", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "orchestrator", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscaler", 2024-07-26T06:18:03.905 INFO:teuthology.orchestra.run.smithi163.stdout: "progress", 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_support", 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "status", 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "telemetry", 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "volumes" 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "last_failure_osd_epoch": 0, 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "active_clients": [ 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "devicehealth", 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.906 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 83055060 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "libcephsqlite", 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2203391701 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.907 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "rbd_support", 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2212531749 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3425233229 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3906161023 2024-07-26T06:18:03.909 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1000525641 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.910 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1386257824 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.911 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2970427520 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3625624196 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1931879051 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "volumes", 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-07-26T06:18:03.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v2", 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.164:0", 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1879372341 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:03.914 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:18:03.919 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr fail x 2024-07-26T06:18:04.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 -- 172.21.15.163:0/1610668887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd015d310 msgr2=0x7f4dd00b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/1610668887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd015d310 0x7f4dd00b4d30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005fd0 tx=0x7f4dc002a540 comp rx=0 tx=0).stop 2024-07-26T06:18:04.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 -- 172.21.15.163:0/1610668887 shutdown_connections 2024-07-26T06:18:04.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/1610668887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd00ce960 0x7f4dd00b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/1610668887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd015d310 0x7f4dd00b4d30 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 -- 172.21.15.163:0/1610668887 >> 172.21.15.163:0/1610668887 conn(0x7f4dd00cfd30 msgr2=0x7f4dd00d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:04.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 -- 172.21.15.163:0/1610668887 shutdown_connections 2024-07-26T06:18:04.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.136+0000 7f4dcfe00640 1 -- 172.21.15.163:0/1610668887 wait complete. 2024-07-26T06:18:04.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.138+0000 7f4dcfe00640 1 Processor -- start 2024-07-26T06:18:04.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.138+0000 7f4dcfe00640 1 -- start start 2024-07-26T06:18:04.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcfe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 0x7f4dd00aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcfe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00ce960 0x7f4dd00ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcfe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd015d310 0x7f4dd00ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcfe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd00b5930 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcfe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd00b57b0 con 0x7f4dd00ce960 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcfe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd00b5ab0 con 0x7f4dd015d310 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 0x7f4dd00aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 0x7f4dd00aa6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50714/0 (socket says 172.21.15.163:50714) 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.139+0000 7f4dcea00640 1 -- 172.21.15.163:0/3817192197 learned_addr learned my addr 172.21.15.163:0/3817192197 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dc7e00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00ce960 0x7f4dd00ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcea00640 1 -- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd015d310 msgr2=0x7f4dd00ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcf400640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd015d310 0x7f4dd00ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcea00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd015d310 0x7f4dd00ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcea00640 1 -- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00ce960 msgr2=0x7f4dd00ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcea00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00ce960 0x7f4dd00ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcea00640 1 -- 172.21.15.163:0/3817192197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd00ab1b0 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcf400640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd015d310 0x7f4dd00ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dc7e00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00ce960 0x7f4dd00ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcea00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 0x7f4dd00aa6b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0034040 tx=0x7f4dc00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc003e030 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4dc00047a0 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd015dde0 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.140+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0035040 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.141+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd015e3a0 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.142+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd00d26c0 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.143+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 12) ==== 98558+0+0 (secure 0 0 0) 0x7f4dc0041030 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.144+0000 7f4dc6a00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4dac0766c0 0x7f4dac078b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.144+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f4dc00b8680 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.144+0000 7f4dc7e00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4dac0766c0 0x7f4dac078b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.146+0000 7f4dc7e00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4dac0766c0 0x7f4dac078b80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0160750 tx=0x7f4db8005db0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:04.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.148+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4dd00d26c0 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.412+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "mgr fail", "who": "x"} v 0) -- 0x7f4dd00cf900 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.421+0000 7f4dc6a00640 1 -- 172.21.15.163:0/3817192197 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "mgr fail", "who": "x"}]=0 v13) ==== 68+0+0 (secure 0 0 0) 0x7f4dd00cf900 con 0x7f4dd00c9ec0 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.424+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4dac0766c0 msgr2=0x7f4dac078b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.424+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4dac0766c0 0x7f4dac078b80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0160750 tx=0x7f4db8005db0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.424+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 msgr2=0x7f4dd00aa6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.424+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 0x7f4dd00aa6b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0034040 tx=0x7f4dc00039e0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.425+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 shutdown_connections 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.425+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:6816/1554374994,v1:172.21.15.164:6817/1554374994] conn(0x7f4dac0766c0 0x7f4dac078b80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.425+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd015d310 0x7f4dd00ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.425+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00ce960 0x7f4dd00ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.425+0000 7f4dcfe00640 1 --2- 172.21.15.163:0/3817192197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4dd00c9ec0 0x7f4dd00aa6b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.425+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 >> 172.21.15.163:0/3817192197 conn(0x7f4dd00cfd30 msgr2=0x7f4dd015c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:04.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.427+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 shutdown_connections 2024-07-26T06:18:04.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.427+0000 7f4dcfe00640 1 -- 172.21.15.163:0/3817192197 wait complete. 2024-07-26T06:18:04.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-07-26T06:18:04.425+0000 7f6d77000640 -1 mgr handle_mgr_map I was active but no longer am 2024-07-26T06:18:04.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:04.551 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.550+0000 7fc4c8400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.551 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.550+0000 7fc4c8400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.551 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.550+0000 7fc4c8400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.551 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.550+0000 7fc4c8400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.551 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.550+0000 7fc4c8400640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.552 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.551+0000 7fc4c4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.552 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.551+0000 7fc4c4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.552 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.551+0000 7fc4c4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.552 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.551+0000 7fc4c4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.552 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:18:04.551+0000 7fc4c4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 -- 172.21.15.163:0/3715792176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c1545e0 msgr2=0x7f299c15b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 --2- 172.21.15.163:0/3715792176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c1545e0 0x7f299c15b830 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f299c0cc0e0 tx=0x7f298c02a560 comp rx=0 tx=0).stop 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 -- 172.21.15.163:0/3715792176 shutdown_connections 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 --2- 172.21.15.163:0/3715792176 >> v2:172.21.15.164:3300/0 conn(0x7f299c06bf40 0x7f299c06c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 --2- 172.21.15.163:0/3715792176 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 0x7f299c0b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 --2- 172.21.15.163:0/3715792176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c1545e0 0x7f299c15b830 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 -- 172.21.15.163:0/3715792176 >> 172.21.15.163:0/3715792176 conn(0x7f299c06d420 msgr2=0x7f299c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 -- 172.21.15.163:0/3715792176 shutdown_connections 2024-07-26T06:18:04.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.660+0000 7f29a1800640 1 -- 172.21.15.163:0/3715792176 wait complete. 2024-07-26T06:18:04.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.661+0000 7f29a1800640 1 Processor -- start 2024-07-26T06:18:04.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.662+0000 7f29a1800640 1 -- start start 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f29a1800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 0x7f299c14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f29a1800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f299c1545e0 0x7f299c14aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f29a1800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 0x7f299c0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f29a1800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f299c072f20 con 0x7f299c06bf40 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f29a1800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f299c072da0 con 0x7f299c1545e0 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f29a1800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f299c0730a0 con 0x7f299c15b390 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 0x7f299c14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 0x7f299c14d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:50816/0 (socket says 172.21.15.163:50816) 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 -- 172.21.15.163:0/47639303 learned_addr learned my addr 172.21.15.163:0/47639303 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299a000640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f299c1545e0 0x7f299c14aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299b400640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 0x7f299c0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 -- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 msgr2=0x7f299c0ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 0x7f299c0ad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 -- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f299c1545e0 msgr2=0x7f299c14aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f299c1545e0 0x7f299c14aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299aa00640 1 -- 172.21.15.163:0/47639303 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c14c860 con 0x7f299c06bf40 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.663+0000 7f299b400640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 0x7f299c0ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.664+0000 7f299aa00640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 0x7f299c14d170 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f298c000c00 tx=0x7f298c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.664+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c03f070 con 0x7f299c06bf40 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.664+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f298c004890 con 0x7f299c06bf40 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.664+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c035030 con 0x7f299c06bf40 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.664+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299c157b20 con 0x7f299c06bf40 2024-07-26T06:18:04.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.664+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f299c14b420 con 0x7f299c06bf40 2024-07-26T06:18:04.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.665+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299c15b830 con 0x7f299c06bf40 2024-07-26T06:18:04.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.665+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 13) ==== 49122+0+0 (secure 0 0 0) 0x7f298c042070 con 0x7f299c06bf40 2024-07-26T06:18:04.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.666+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f298c072a50 con 0x7f299c06bf40 2024-07-26T06:18:04.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.670+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f298c004f30 con 0x7f299c06bf40 2024-07-26T06:18:04.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.907+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299c0af170 con 0x7f299c06bf40 2024-07-26T06:18:04.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.908+0000 7f2992a00640 1 -- 172.21.15.163:0/47639303 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7f298c042a50 con 0x7f299c06bf40 2024-07-26T06:18:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:04.909 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:04.910 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:04.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 msgr2=0x7f299c14d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 0x7f299c14d170 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f298c000c00 tx=0x7f298c0039e0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 shutdown_connections 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f299c15b390 0x7f299c0ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f299c1545e0 0x7f299c14aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 --2- 172.21.15.163:0/47639303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f299c06bf40 0x7f299c14d170 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 >> 172.21.15.163:0/47639303 conn(0x7f299c06d420 msgr2=0x7f299c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 shutdown_connections 2024-07-26T06:18:04.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:04.911+0000 7f29a1800640 1 -- 172.21.15.163:0/47639303 wait complete. 2024-07-26T06:18:04.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-07-26T06:18:05.145 INFO:teuthology.orchestra.run.smithi163.stdout:[{"id":5102,"entity":{"name":{"type":"client","num":5102},"addr":{"type":"any","addr":"172.21.15.163:0","nonce":1704589760}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.57398726799999999,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9973655253378988,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5102 172.21.15.163:0/1704589760","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"y","hostname":"smithi163","pid":"37377","root":"/"}},{"id":5356,"entity":{"name":{"type":"client","num":5356},"addr":{"type":"any","addr":"172.21.15.163:0","nonce":821490554}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.57498724599999995,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9973655253378988,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5356 172.21.15.163:0/821490554","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4aae82f07003420ccc152fe0b01f3d5a44885708","ceph_version":"ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)","entity_id":"y","hostname":"smithi163","pid":"37377","root":"/"}}] 2024-07-26T06:18:05.145 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi163", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37377", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.163:0", "nonce": 1704589760, "type": "any" }, "name": { "num": 5102, "type": "client" } }, "id": 5102, "inst": "client.5102 172.21.15.163:0/1704589760", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9973655253378988 }, "state": "open", "uptime": 0.573987268 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4aae82f07003420ccc152fe0b01f3d5a44885708", "ceph_version": "ceph version 19.0.0-4588-g4aae82f0 (4aae82f07003420ccc152fe0b01f3d5a44885708) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi163", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "37377", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.163:0", "nonce": 821490554, "type": "any" }, "name": { "num": 5356, "type": "client" } }, "id": 5356, "inst": "client.5356 172.21.15.163:0/821490554", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9973655253378988 }, "state": "open", "uptime": 0.574987246 } ] 2024-07-26T06:18:05.146 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-07-26T06:18:05.146 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:18:05.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:05.273 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:05.387 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:18:05.387 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:05.387 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:05.387 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:05.450 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:05.450 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:05.450 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:05.450 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:05.498 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:05.499 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:05.499 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:05.499 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:05.554 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:05.554 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:05.554 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:05.554 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:05.598 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:05.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 -- 172.21.15.163:0/1249867281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 msgr2=0x7f6fc00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 --2- 172.21.15.163:0/1249867281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00ce2f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0160e50 tx=0x7f6fa802a560 comp rx=0 tx=0).stop 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 -- 172.21.15.163:0/1249867281 shutdown_connections 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 --2- 172.21.15.163:0/1249867281 >> v2:172.21.15.164:3300/0 conn(0x7f6fc00ce830 0x7f6fc00b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 --2- 172.21.15.163:0/1249867281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00ce2f0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 -- 172.21.15.163:0/1249867281 >> 172.21.15.163:0/1249867281 conn(0x7f6fc00cfda0 msgr2=0x7f6fc00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:05.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 -- 172.21.15.163:0/1249867281 shutdown_connections 2024-07-26T06:18:05.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.806+0000 7f6fc6600640 1 -- 172.21.15.163:0/1249867281 wait complete. 2024-07-26T06:18:05.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.808+0000 7f6fc6600640 1 Processor -- start 2024-07-26T06:18:05.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.808+0000 7f6fc6600640 1 -- start start 2024-07-26T06:18:05.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.809+0000 7f6fc6600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00b1e80 0x7f6fc0150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:05.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.809+0000 7f6fc6600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc00ce830 0x7f6fc0152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:05.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.809+0000 7f6fc6600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:05.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.809+0000 7f6fc6600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc00b5130 con 0x7f6fc015d5a0 2024-07-26T06:18:05.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.809+0000 7f6fc6600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc00b4fb0 con 0x7f6fc00b1e80 2024-07-26T06:18:05.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.809+0000 7f6fc6600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc00b52b0 con 0x7f6fc00ce830 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43218/0 (socket says 172.21.15.163:43218) 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 -- 172.21.15.163:0/535532145 learned_addr learned my addr 172.21.15.163:0/535532145 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fbfe00640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc00ce830 0x7f6fc0152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5200640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00b1e80 0x7f6fc0150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 -- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc00ce830 msgr2=0x7f6fc0152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc00ce830 0x7f6fc0152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 -- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00b1e80 msgr2=0x7f6fc0150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00b1e80 0x7f6fc0150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5c00640 1 -- 172.21.15.163:0/535532145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc0152c60 con 0x7f6fc015d5a0 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.810+0000 7f6fc5200640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00b1e80 0x7f6fc0150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.811+0000 7f6fc5c00640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00cc8b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800c400 tx=0x7f6fb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:05.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.811+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8017070 con 0x7f6fc015d5a0 2024-07-26T06:18:05.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.811+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fb80053c0 con 0x7f6fc015d5a0 2024-07-26T06:18:05.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.811+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8003850 con 0x7f6fc015d5a0 2024-07-26T06:18:05.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.811+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc0152f40 con 0x7f6fc015d5a0 2024-07-26T06:18:05.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.811+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc0153500 con 0x7f6fc015d5a0 2024-07-26T06:18:05.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.812+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc0074680 con 0x7f6fc015d5a0 2024-07-26T06:18:05.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.812+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 14) ==== 49249+0+0 (secure 0 0 0) 0x7f6fb801a070 con 0x7f6fc015d5a0 2024-07-26T06:18:05.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.813+0000 7f6fbcc00640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6f9c03d210 0x7f6f9c03f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:05.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.813+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f6fb8051090 con 0x7f6fc015d5a0 2024-07-26T06:18:05.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.813+0000 7f6fc5200640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6f9c03d210 0x7f6f9c03f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:05.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.814+0000 7f6fc5200640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6f9c03d210 0x7f6f9c03f6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8033000 tx=0x7f6fa8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:05.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:05.818+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6fb80102e0 con 0x7f6fc015d5a0 2024-07-26T06:18:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.058+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fc00b5820 con 0x7f6fc015d5a0 2024-07-26T06:18:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.059+0000 7f6fbcc00640 1 -- 172.21.15.163:0/535532145 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7f6fb801bdf0 con 0x7f6fc015d5a0 2024-07-26T06:18:06.060 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:06.060 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6f9c03d210 msgr2=0x7f6f9c03f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6f9c03d210 0x7f6f9c03f6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8033000 tx=0x7f6fa8004730 comp rx=0 tx=0).stop 2024-07-26T06:18:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 msgr2=0x7f6fc00cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00cc8b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800c400 tx=0x7f6fb800c8c0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 shutdown_connections 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6f9c03d210 0x7f6f9c03f6d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6fc015d5a0 0x7f6fc00cc8b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc00ce830 0x7f6fc0152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 --2- 172.21.15.163:0/535532145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00b1e80 0x7f6fc0150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.062+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 >> 172.21.15.163:0/535532145 conn(0x7f6fc00cfda0 msgr2=0x7f6fc015b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.063+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 shutdown_connections 2024-07-26T06:18:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.063+0000 7f6fc6600640 1 -- 172.21.15.163:0/535532145 wait complete. 2024-07-26T06:18:06.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.273+0000 7f258f200640 1 -- 172.21.15.163:0/2744065553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25901556f0 msgr2=0x7f25900cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.273+0000 7f258f200640 1 --2- 172.21.15.163:0/2744065553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25901556f0 0x7f25900cdd90 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f259006cf00 tx=0x7f258002a530 comp rx=0 tx=0).stop 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.273+0000 7f258f200640 1 -- 172.21.15.163:0/2744065553 shutdown_connections 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.273+0000 7f258f200640 1 --2- 172.21.15.163:0/2744065553 >> v2:172.21.15.164:3300/0 conn(0x7f25900ce2d0 0x7f2590071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.273+0000 7f258f200640 1 --2- 172.21.15.163:0/2744065553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25901556f0 0x7f25900cdd90 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.273+0000 7f258f200640 1 -- 172.21.15.163:0/2744065553 >> 172.21.15.163:0/2744065553 conn(0x7f25900b0e30 msgr2=0x7f25900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.274+0000 7f258f200640 1 -- 172.21.15.163:0/2744065553 shutdown_connections 2024-07-26T06:18:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.274+0000 7f258f200640 1 -- 172.21.15.163:0/2744065553 wait complete. 2024-07-26T06:18:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.275+0000 7f258f200640 1 Processor -- start 2024-07-26T06:18:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.275+0000 7f258f200640 1 -- start start 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 0x7f25901411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25900ce2d0 0x7f25901e92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25901556f0 0x7f25901e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2590072000 con 0x7f2590071a70 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2590071e80 con 0x7f25901556f0 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f259015f310 con 0x7f25900ce2d0 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 0x7f25901411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 0x7f25901411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43230/0 (socket says 172.21.15.163:43230) 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.276+0000 7f258de00640 1 -- 172.21.15.163:0/3790792033 learned_addr learned my addr 172.21.15.163:0/3790792033 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258d400640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25900ce2d0 0x7f25901e92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258e800640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25901556f0 0x7f25901e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258de00640 1 -- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25900ce2d0 msgr2=0x7f25901e92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258de00640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25900ce2d0 0x7f25901e92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258de00640 1 -- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25901556f0 msgr2=0x7f25901e62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258de00640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25901556f0 0x7f25901e62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258de00640 1 -- 172.21.15.163:0/3790792033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25901e6d80 con 0x7f2590071a70 2024-07-26T06:18:06.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f258de00640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 0x7f25901411a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2580009780 tx=0x7f2580033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.277+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580031d60 con 0x7f2590071a70 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.278+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2580005020 con 0x7f2590071a70 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.278+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25901e7060 con 0x7f2590071a70 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.278+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580005470 con 0x7f2590071a70 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.278+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25901e7620 con 0x7f2590071a70 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.279+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f259015f850 con 0x7f2590071a70 2024-07-26T06:18:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.279+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 14) ==== 49249+0+0 (secure 0 0 0) 0x7f25800047a0 con 0x7f2590071a70 2024-07-26T06:18:06.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.279+0000 7f2586000640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f257003d210 0x7f257003f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.279+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f2580072f20 con 0x7f2590071a70 2024-07-26T06:18:06.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.279+0000 7f258d400640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f257003d210 0x7f257003f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.280+0000 7f258d400640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f257003d210 0x7f257003f6d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f25901f76c0 tx=0x7f257c008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:06.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.284+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f258003f030 con 0x7f2590071a70 2024-07-26T06:18:06.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.439+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 15) ==== 49457+0+0 (secure 0 0 0) 0x7f2580033070 con 0x7f2590071a70 2024-07-26T06:18:06.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.535+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25900b34f0 con 0x7f2590071a70 2024-07-26T06:18:06.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.536+0000 7f2586000640 1 -- 172.21.15.163:0/3790792033 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7f2580005610 con 0x7f2590071a70 2024-07-26T06:18:06.537 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:06.537 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:06.537 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.538+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f257003d210 msgr2=0x7f257003f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.538+0000 7f258f200640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f257003d210 0x7f257003f6d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f25901f76c0 tx=0x7f257c008040 comp rx=0 tx=0).stop 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 msgr2=0x7f25901411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 0x7f25901411a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2580009780 tx=0x7f2580033d60 comp rx=0 tx=0).stop 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 shutdown_connections 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f257003d210 0x7f257003f6d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25901556f0 0x7f25901e62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25900ce2d0 0x7f25901e92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 --2- 172.21.15.163:0/3790792033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2590071a70 0x7f25901411a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 >> 172.21.15.163:0/3790792033 conn(0x7f25900b0e30 msgr2=0x7f259015d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:06.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 shutdown_connections 2024-07-26T06:18:06.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.539+0000 7f258f200640 1 -- 172.21.15.163:0/3790792033 wait complete. 2024-07-26T06:18:06.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.740+0000 7f4567c00640 1 -- 172.21.15.163:0/2116022932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 msgr2=0x7f456015be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.740+0000 7f4567c00640 1 --2- 172.21.15.163:0/2116022932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f456015be40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4560070bc0 tx=0x7f454c02a4d0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 -- 172.21.15.163:0/2116022932 shutdown_connections 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 --2- 172.21.15.163:0/2116022932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456015c380 0x7f456006bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 --2- 172.21.15.163:0/2116022932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f456015be40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 --2- 172.21.15.163:0/2116022932 >> v2:172.21.15.164:3300/0 conn(0x7f45600cb7e0 0x7f456015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 -- 172.21.15.163:0/2116022932 >> 172.21.15.163:0/2116022932 conn(0x7f456006d420 msgr2=0x7f456006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:06.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 -- 172.21.15.163:0/2116022932 shutdown_connections 2024-07-26T06:18:06.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.741+0000 7f4567c00640 1 -- 172.21.15.163:0/2116022932 wait complete. 2024-07-26T06:18:06.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.742+0000 7f4567c00640 1 Processor -- start 2024-07-26T06:18:06.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.742+0000 7f4567c00640 1 -- start start 2024-07-26T06:18:06.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.743+0000 7f4567c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f45600cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.743+0000 7f4567c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45600cb7e0 0x7f45600ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.743+0000 7f4567c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456015c380 0x7f4560159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.743+0000 7f4567c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f456006c810 con 0x7f4560071cf0 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.743+0000 7f4567c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f456006c690 con 0x7f456015c380 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.743+0000 7f4567c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4560073530 con 0x7f45600cb7e0 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f45600cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f45600cdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43254/0 (socket says 172.21.15.163:43254) 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 -- 172.21.15.163:0/84015830 learned_addr learned my addr 172.21.15.163:0/84015830 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456015c380 0x7f4560159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f455fe00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45600cb7e0 0x7f45600ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 -- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45600cb7e0 msgr2=0x7f45600ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45600cb7e0 0x7f45600ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 -- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456015c380 msgr2=0x7f4560159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456015c380 0x7f4560159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 -- 172.21.15.163:0/84015830 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45601f7400 con 0x7f4560071cf0 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f455fe00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45600cb7e0 0x7f45600ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.744+0000 7f4565200640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f45600cdf80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f455400d7a0 tx=0x7f455400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.745+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4554017070 con 0x7f4560071cf0 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.745+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4554009d20 con 0x7f4560071cf0 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.745+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45601f76e0 con 0x7f4560071cf0 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.745+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45540106c0 con 0x7f4560071cf0 2024-07-26T06:18:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.745+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45601f7ca0 con 0x7f4560071cf0 2024-07-26T06:18:06.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.746+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4560072860 con 0x7f4560071cf0 2024-07-26T06:18:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.746+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 49457+0+0 (secure 0 0 0) 0x7f455401a070 con 0x7f4560071cf0 2024-07-26T06:18:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.747+0000 7f455cc00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f454003d480 0x7f454003f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.747+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f4554052380 con 0x7f4560071cf0 2024-07-26T06:18:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.747+0000 7f455fe00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f454003d480 0x7f454003f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.748+0000 7f455fe00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f454003d480 0x7f454003f940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f454c033000 tx=0x7f454c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.752+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4560072860 con 0x7f4560071cf0 2024-07-26T06:18:06.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.955+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45600af190 con 0x7f4560071cf0 2024-07-26T06:18:06.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.958+0000 7f455cc00640 1 -- 172.21.15.163:0/84015830 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+17514 (secure 0 0 0) 0x7f45540520d0 con 0x7f4560071cf0 2024-07-26T06:18:06.959 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:06.959 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":70,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:04.414758+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:44.680290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:45.272282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:0/2203391701":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1931879051":"2024-07-27T06:18:04.414635+0000","172.21.15.164:6817/1554374994":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1386257824":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3425233229":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/83055060":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3906161023":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1000525641":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3625624196":"2024-07-27T06:18:04.414635+0000","172.21.15.164:6816/1554374994":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1879372341":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/2970427520":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/2212531749":"2024-07-27T06:18:04.414635+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:06.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f454003d480 msgr2=0x7f454003f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f454003d480 0x7f454003f940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f454c033000 tx=0x7f454c004730 comp rx=0 tx=0).stop 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 msgr2=0x7f45600cdf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f45600cdf80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f455400d7a0 tx=0x7f455400dc60 comp rx=0 tx=0).stop 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 shutdown_connections 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f454003d480 0x7f454003f940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456015c380 0x7f4560159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45600cb7e0 0x7f45600ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 --2- 172.21.15.163:0/84015830 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4560071cf0 0x7f45600cdf80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:06.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 >> 172.21.15.163:0/84015830 conn(0x7f456006d420 msgr2=0x7f45600b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:06.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 shutdown_connections 2024-07-26T06:18:06.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:06.961+0000 7f4567c00640 1 -- 172.21.15.163:0/84015830 wait complete. 2024-07-26T06:18:06.981 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:18:06.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 -- 172.21.15.163:0/1837707057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc70004800 msgr2=0x7fcc70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 --2- 172.21.15.163:0/1837707057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc70004800 0x7fcc70004c80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400c310 tx=0x7fcc70005830 comp rx=0 tx=0).stop 2024-07-26T06:18:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 -- 172.21.15.163:0/1837707057 shutdown_connections 2024-07-26T06:18:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 --2- 172.21.15.163:0/1837707057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc70004800 0x7fcc70004c80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 -- 172.21.15.163:0/1837707057 >> 172.21.15.163:0/1837707057 conn(0x7fcc8c0b0e30 msgr2=0x7fcc8c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 -- 172.21.15.163:0/1837707057 shutdown_connections 2024-07-26T06:18:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.178+0000 7fcc8be00640 1 -- 172.21.15.163:0/1837707057 wait complete. 2024-07-26T06:18:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.180+0000 7fcc8be00640 1 Processor -- start 2024-07-26T06:18:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.180+0000 7fcc8be00640 1 -- start start 2024-07-26T06:18:07.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc70004800 0x7fcc8c1f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc8c147d30 0x7fcc8c1e3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 0x7fcc8c1475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c0718c0 con 0x7fcc8c147100 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c071740 con 0x7fcc70004800 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcc8c071a40 con 0x7fcc8c147d30 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 0x7fcc8c1475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 0x7fcc8c1475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43272/0 (socket says 172.21.15.163:43272) 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8b400640 1 -- 172.21.15.163:0/2861384708 learned_addr learned my addr 172.21.15.163:0/2861384708 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc8aa00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc70004800 0x7fcc8c1f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.181+0000 7fcc82000640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc8c147d30 0x7fcc8c1e3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8b400640 1 -- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc8c147d30 msgr2=0x7fcc8c1e3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8b400640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc8c147d30 0x7fcc8c1e3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8b400640 1 -- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc70004800 msgr2=0x7fcc8c1f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8b400640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc70004800 0x7fcc8c1f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8b400640 1 -- 172.21.15.163:0/2861384708 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8c147af0 con 0x7fcc8c147100 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8b400640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 0x7fcc8c1475b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcc84005fd0 tx=0x7fcc8400fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc84003d80 con 0x7fcc8c147100 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc84017070 con 0x7fcc8c147100 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8400dd10 con 0x7fcc8c147100 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.182+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc8c1484b0 con 0x7fcc8c147100 2024-07-26T06:18:07.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.183+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc8c148a70 con 0x7fcc8c147100 2024-07-26T06:18:07.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.184+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 49457+0+0 (secure 0 0 0) 0x7fcc84004030 con 0x7fcc8c147100 2024-07-26T06:18:07.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.184+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc8c0cdf50 con 0x7fcc8c147100 2024-07-26T06:18:07.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.184+0000 7fcc83e00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcc6803d450 0x7fcc6803f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.184+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7fcc84052a80 con 0x7fcc8c147100 2024-07-26T06:18:07.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.184+0000 7fcc8aa00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcc6803d450 0x7fcc6803f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.185+0000 7fcc8aa00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcc6803d450 0x7fcc6803f910 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c006630 tx=0x7fcc7c006310 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:07.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.189+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc84004320 con 0x7fcc8c147100 2024-07-26T06:18:07.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.437+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc8c0ced60 con 0x7fcc8c147100 2024-07-26T06:18:07.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.438+0000 7fcc83e00640 1 -- 172.21.15.163:0/2861384708 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7fcc84020da0 con 0x7fcc8c147100 2024-07-26T06:18:07.440 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:07.440 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:07.441 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.441+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcc6803d450 msgr2=0x7fcc6803f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.441+0000 7fcc8be00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcc6803d450 0x7fcc6803f910 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c006630 tx=0x7fcc7c006310 comp rx=0 tx=0).stop 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 msgr2=0x7fcc8c1475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 0x7fcc8c1475b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcc84005fd0 tx=0x7fcc8400fef0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 shutdown_connections 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcc6803d450 0x7fcc6803f910 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc8c147100 0x7fcc8c1475b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc8c147d30 0x7fcc8c1e3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 --2- 172.21.15.163:0/2861384708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc70004800 0x7fcc8c1f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 >> 172.21.15.163:0/2861384708 conn(0x7fcc8c0b0e30 msgr2=0x7fcc8c150100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 shutdown_connections 2024-07-26T06:18:07.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.442+0000 7fcc8be00640 1 -- 172.21.15.163:0/2861384708 wait complete. 2024-07-26T06:18:07.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.658+0000 7f7321200640 1 -- 172.21.15.163:0/2855376141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0b4420 msgr2=0x7f731c0b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.658+0000 7f7321200640 1 --2- 172.21.15.163:0/2855376141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0b4420 0x7f731c0b2130 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f731c0b1ac0 tx=0x7f730802a540 comp rx=0 tx=0).stop 2024-07-26T06:18:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.659+0000 7f7321200640 1 -- 172.21.15.163:0/2855376141 shutdown_connections 2024-07-26T06:18:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.659+0000 7f7321200640 1 --2- 172.21.15.163:0/2855376141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0b4420 0x7f731c0b2130 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.659+0000 7f7321200640 1 --2- 172.21.15.163:0/2855376141 >> v2:172.21.15.164:3300/0 conn(0x7f731c0cdc40 0x7f731c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.659+0000 7f7321200640 1 -- 172.21.15.163:0/2855376141 >> 172.21.15.163:0/2855376141 conn(0x7f731c0b0e30 msgr2=0x7f731c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.659+0000 7f7321200640 1 -- 172.21.15.163:0/2855376141 shutdown_connections 2024-07-26T06:18:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.659+0000 7f7321200640 1 -- 172.21.15.163:0/2855376141 wait complete. 2024-07-26T06:18:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.660+0000 7f7321200640 1 Processor -- start 2024-07-26T06:18:07.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.661+0000 7f7321200640 1 -- start start 2024-07-26T06:18:07.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.661+0000 7f7321200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f731c0b4420 0x7f731c1f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f7321200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 0x7f731c073240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f7321200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f731c1e56e0 0x7f731c15b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f7321200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f731c06f500 con 0x7f731c0cdc40 2024-07-26T06:18:07.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f7321200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f731c0b2bb0 con 0x7f731c1e56e0 2024-07-26T06:18:07.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f7321200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f731c1e7080 con 0x7f731c0b4420 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 0x7f731c073240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 0x7f731c073240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43290/0 (socket says 172.21.15.163:43290) 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 -- 172.21.15.163:0/3070560415 learned_addr learned my addr 172.21.15.163:0/3070560415 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731be00640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f731c1e56e0 0x7f731c15b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731b400640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f731c0b4420 0x7f731c1f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 -- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f731c0b4420 msgr2=0x7f731c1f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f731c0b4420 0x7f731c1f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 -- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f731c1e56e0 msgr2=0x7f731c15b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f731c1e56e0 0x7f731c15b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731aa00640 1 -- 172.21.15.163:0/3070560415 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731c1e72b0 con 0x7f731c0cdc40 2024-07-26T06:18:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.662+0000 7f731b400640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f731c0b4420 0x7f731c1f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:07.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.663+0000 7f731aa00640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 0x7f731c073240 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7308009780 tx=0x7f7308032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:07.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.663+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7308040070 con 0x7f731c0cdc40 2024-07-26T06:18:07.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.663+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7308005310 con 0x7f731c0cdc40 2024-07-26T06:18:07.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.663+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f731c1e7590 con 0x7f731c0cdc40 2024-07-26T06:18:07.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.663+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73080047a0 con 0x7f731c0cdc40 2024-07-26T06:18:07.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.663+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f731c1e5b90 con 0x7f731c0cdc40 2024-07-26T06:18:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.664+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 49457+0+0 (secure 0 0 0) 0x7f7308039040 con 0x7f731c0cdc40 2024-07-26T06:18:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.664+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f731c0cd530 con 0x7f731c0cdc40 2024-07-26T06:18:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.665+0000 7f7313400640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f72fc03d450 0x7f72fc03f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.665+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f73080735d0 con 0x7f731c0cdc40 2024-07-26T06:18:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.665+0000 7f731b400640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f72fc03d450 0x7f72fc03f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.666+0000 7f731b400640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f72fc03d450 0x7f72fc03f910 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f730c007fd0 tx=0x7f730c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:07.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.670+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7308004de0 con 0x7f731c0cdc40 2024-07-26T06:18:07.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.916+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f731c0b0070 con 0x7f731c0cdc40 2024-07-26T06:18:07.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.917+0000 7f7313400640 1 -- 172.21.15.163:0/3070560415 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5237 (secure 0 0 0) 0x7f73080035f0 con 0x7f731c0cdc40 2024-07-26T06:18:07.918 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:07.918 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":46,"btime":"2024-07-26T06:17:57:922612+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:17:46.285783+0000","modified":"2024-07-26T06:17:49.380437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6090},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6090":{"gid":6090,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6835/1648150930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1648150930},{"type":"v1","addr":"172.21.15.163:6835","nonce":1648150930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6090,"qdb_cluster":[6090]},"id":5}]} 2024-07-26T06:18:07.918 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 46 2024-07-26T06:18:07.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.919+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f72fc03d450 msgr2=0x7f72fc03f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.919+0000 7f7321200640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f72fc03d450 0x7f72fc03f910 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f730c007fd0 tx=0x7f730c009040 comp rx=0 tx=0).stop 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.919+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 msgr2=0x7f731c073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.919+0000 7f7321200640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 0x7f731c073240 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7308009780 tx=0x7f7308032c10 comp rx=0 tx=0).stop 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 shutdown_connections 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f72fc03d450 0x7f72fc03f910 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f731c1e56e0 0x7f731c15b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f731c0cdc40 0x7f731c073240 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 --2- 172.21.15.163:0/3070560415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f731c0b4420 0x7f731c1f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 >> 172.21.15.163:0/3070560415 conn(0x7f731c0b0e30 msgr2=0x7f731c159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 shutdown_connections 2024-07-26T06:18:07.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:07.920+0000 7f7321200640 1 -- 172.21.15.163:0/3070560415 wait complete. 2024-07-26T06:18:07.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:08.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 -- 172.21.15.163:0/1553960809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c680ade20 msgr2=0x7f7c680cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 --2- 172.21.15.163:0/1553960809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c680ade20 0x7f7c680cbd40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7c68075c30 tx=0x7f7c5402a540 comp rx=0 tx=0).stop 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 -- 172.21.15.163:0/1553960809 shutdown_connections 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 --2- 172.21.15.163:0/1553960809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c680ade20 0x7f7c680cbd40 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 --2- 172.21.15.163:0/1553960809 >> v2:172.21.15.164:3300/0 conn(0x7f7c68146be0 0x7f7c680ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 -- 172.21.15.163:0/1553960809 >> 172.21.15.163:0/1553960809 conn(0x7f7c680b0e30 msgr2=0x7f7c680b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.149+0000 7f7c67e00640 1 -- 172.21.15.163:0/1553960809 shutdown_connections 2024-07-26T06:18:08.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.150+0000 7f7c67e00640 1 -- 172.21.15.163:0/1553960809 wait complete. 2024-07-26T06:18:08.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.151+0000 7f7c67e00640 1 Processor -- start 2024-07-26T06:18:08.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.151+0000 7f7c67e00640 1 -- start start 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.152+0000 7f7c67e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c680ade20 0x7f7c6813ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.152+0000 7f7c67e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c680cc280 0x7f7c68071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.152+0000 7f7c67e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 0x7f7c681459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.152+0000 7f7c67e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c680b48e0 con 0x7f7c68146be0 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.152+0000 7f7c67e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7c680b4760 con 0x7f7c680cc280 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.152+0000 7f7c67e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c680ce520 con 0x7f7c680ade20 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 0x7f7c681459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 0x7f7c681459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43312/0 (socket says 172.21.15.163:43312) 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 -- 172.21.15.163:0/540497688 learned_addr learned my addr 172.21.15.163:0/540497688 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c66000640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c680cc280 0x7f7c68071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c66a00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c680ade20 0x7f7c6813ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 -- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c680ade20 msgr2=0x7f7c6813ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c680ade20 0x7f7c6813ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 -- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c680cc280 msgr2=0x7f7c68071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c680cc280 0x7f7c68071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c67400640 1 -- 172.21.15.163:0/540497688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6813f4c0 con 0x7f7c68146be0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.153+0000 7f7c66000640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c680cc280 0x7f7c68071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.154+0000 7f7c67400640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 0x7f7c681459b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000cbc0 tx=0x7f7c6000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.154+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60016020 con 0x7f7c68146be0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.154+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c6000add0 con 0x7f7c68146be0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.154+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60012680 con 0x7f7c68146be0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.154+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c68072050 con 0x7f7c68146be0 2024-07-26T06:18:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.154+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c68072610 con 0x7f7c68146be0 2024-07-26T06:18:08.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.155+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c680cea20 con 0x7f7c68146be0 2024-07-26T06:18:08.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.155+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 49457+0+0 (secure 0 0 0) 0x7f7c60019020 con 0x7f7c68146be0 2024-07-26T06:18:08.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.156+0000 7f7c5ea00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c4803d480 0x7f7c4803f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.156+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f7c600526d0 con 0x7f7c68146be0 2024-07-26T06:18:08.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.156+0000 7f7c66a00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c4803d480 0x7f7c4803f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.157+0000 7f7c66a00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c4803d480 0x7f7c4803f940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c580058c0 tx=0x7f7c58005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:08.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.161+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c60020020 con 0x7f7c68146be0 2024-07-26T06:18:08.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.371+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c680ae8b0 con 0x7f7c68146be0 2024-07-26T06:18:08.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.373+0000 7f7c5ea00640 1 -- 172.21.15.163:0/540497688 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+17514 (secure 0 0 0) 0x7f7c60010cf0 con 0x7f7c68146be0 2024-07-26T06:18:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:08.374 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":70,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:04.414758+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:17:44.680290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-07-26T06:17:45.272282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:0/2203391701":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1931879051":"2024-07-27T06:18:04.414635+0000","172.21.15.164:6817/1554374994":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1386257824":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3425233229":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/83055060":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3906161023":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1000525641":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3625624196":"2024-07-27T06:18:04.414635+0000","172.21.15.164:6816/1554374994":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1879372341":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/2970427520":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/2212531749":"2024-07-27T06:18:04.414635+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:08.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c4803d480 msgr2=0x7f7c4803f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c4803d480 0x7f7c4803f940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c580058c0 tx=0x7f7c58005850 comp rx=0 tx=0).stop 2024-07-26T06:18:08.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 msgr2=0x7f7c681459b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 0x7f7c681459b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000cbc0 tx=0x7f7c6000a430 comp rx=0 tx=0).stop 2024-07-26T06:18:08.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 shutdown_connections 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c4803d480 0x7f7c4803f940 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c68146be0 0x7f7c681459b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c680cc280 0x7f7c68071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 --2- 172.21.15.163:0/540497688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c680ade20 0x7f7c6813ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 >> 172.21.15.163:0/540497688 conn(0x7f7c680b0e30 msgr2=0x7f7c680cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 shutdown_connections 2024-07-26T06:18:08.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.376+0000 7f7c67e00640 1 -- 172.21.15.163:0/540497688 wait complete. 2024-07-26T06:18:08.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:18:08.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.593+0000 7f5318a00640 1 -- 172.21.15.163:0/2203036980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140adff0 msgr2=0x7f53140cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.593+0000 7f5318a00640 1 --2- 172.21.15.163:0/2203036980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140adff0 0x7f53140cbd40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f531406f710 tx=0x7f52fc02a540 comp rx=0 tx=0).stop 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.594+0000 7f5318a00640 1 -- 172.21.15.163:0/2203036980 shutdown_connections 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.594+0000 7f5318a00640 1 --2- 172.21.15.163:0/2203036980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140adff0 0x7f53140cbd40 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.594+0000 7f5318a00640 1 --2- 172.21.15.163:0/2203036980 >> v2:172.21.15.164:3300/0 conn(0x7f53140cc280 0x7f53140b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.594+0000 7f5318a00640 1 -- 172.21.15.163:0/2203036980 >> 172.21.15.163:0/2203036980 conn(0x7f53140b0e30 msgr2=0x7f53140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.594+0000 7f5318a00640 1 -- 172.21.15.163:0/2203036980 shutdown_connections 2024-07-26T06:18:08.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.594+0000 7f5318a00640 1 -- 172.21.15.163:0/2203036980 wait complete. 2024-07-26T06:18:08.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.595+0000 7f5318a00640 1 Processor -- start 2024-07-26T06:18:08.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.595+0000 7f5318a00640 1 -- start start 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f5318a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53140adff0 0x7f531413cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f5318a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 0x7f5314138a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f5318a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5314146be0 0x7f5314138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f5318a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53140b4020 con 0x7f53140cc280 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f5318a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53140b3ea0 con 0x7f53140adff0 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f5318a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53140b41a0 con 0x7f5314146be0 2024-07-26T06:18:08.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f530aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 0x7f5314138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f530aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 0x7f5314138a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43322/0 (socket says 172.21.15.163:43322) 2024-07-26T06:18:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.596+0000 7f530aa00640 1 -- 172.21.15.163:0/3273666499 learned_addr learned my addr 172.21.15.163:0/3273666499 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f5313e00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5314146be0 0x7f5314138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f5313400640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53140adff0 0x7f531413cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530aa00640 1 -- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5314146be0 msgr2=0x7f5314138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530aa00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5314146be0 0x7f5314138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530aa00640 1 -- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53140adff0 msgr2=0x7f531413cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530aa00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53140adff0 0x7f531413cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530aa00640 1 -- 172.21.15.163:0/3273666499 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531413eb70 con 0x7f53140cc280 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f5313e00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5314146be0 0x7f5314138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530aa00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 0x7f5314138a90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009920 tx=0x7f52fc032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc040070 con 0x7f53140cc280 2024-07-26T06:18:08.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52fc005310 con 0x7f53140cc280 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc003800 con 0x7f53140cc280 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.597+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531413edf0 con 0x7f53140cc280 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.598+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f531413f3b0 con 0x7f53140cc280 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.599+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 49513+0+0 (secure 0 0 0) 0x7f52fc039040 con 0x7f53140cc280 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.599+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53140b4700 con 0x7f53140cc280 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.599+0000 7f530be00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f52f003d4e0 0x7f52f003f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:08.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.599+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6268+0+0 (secure 0 0 0) 0x7f52fc0735f0 con 0x7f53140cc280 2024-07-26T06:18:08.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.599+0000 7f5313400640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f52f003d4e0 0x7f52f003f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:08.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.600+0000 7f5313400640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f52f003d4e0 0x7f52f003f9a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5304001540 tx=0x7f5304009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:08.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.604+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f531413eb70 con 0x7f53140cc280 2024-07-26T06:18:08.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.836+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f53140ae8b0 con 0x7f53140cc280 2024-07-26T06:18:08.838 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:18:08.838+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:18:08.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.842+0000 7f530be00640 1 -- 172.21.15.163:0/3273666499 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) ==== 161+0+0 (secure 0 0 0) 0x7f52fc003a50 con 0x7f53140cc280 2024-07-26T06:18:08.843 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f52f003d4e0 msgr2=0x7f52f003f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f52f003d4e0 0x7f52f003f9a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5304001540 tx=0x7f5304009040 comp rx=0 tx=0).stop 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 msgr2=0x7f5314138a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 0x7f5314138a90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009920 tx=0x7f52fc032d50 comp rx=0 tx=0).stop 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 shutdown_connections 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f52f003d4e0 0x7f52f003f9a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5314146be0 0x7f5314138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53140cc280 0x7f5314138a90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 --2- 172.21.15.163:0/3273666499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53140adff0 0x7f531413cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:08.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 >> 172.21.15.163:0/3273666499 conn(0x7f53140b0e30 msgr2=0x7f53140b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:08.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 shutdown_connections 2024-07-26T06:18:08.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:08.845+0000 7f5318a00640 1 -- 172.21.15.163:0/3273666499 wait complete. 2024-07-26T06:18:08.863 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:18:08.881 INFO:tasks.ceph.mds.c.smithi163.stderr: -8067> 2024-07-26T06:17:48.751+0000 7fa392600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:18:08.968 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:18:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 -- 172.21.15.163:0/108899332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 msgr2=0x7fb1c40744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 --2- 172.21.15.163:0/108899332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c40744b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4005fd0 tx=0x7fb1b402a540 comp rx=0 tx=0).stop 2024-07-26T06:18:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 -- 172.21.15.163:0/108899332 shutdown_connections 2024-07-26T06:18:09.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 --2- 172.21.15.163:0/108899332 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c40744b0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 --2- 172.21.15.163:0/108899332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c41413b0 0x7fb1c415b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 -- 172.21.15.163:0/108899332 >> 172.21.15.163:0/108899332 conn(0x7fb1c406bf40 msgr2=0x7fb1c406c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 -- 172.21.15.163:0/108899332 shutdown_connections 2024-07-26T06:18:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.036+0000 7fb1ca600640 1 -- 172.21.15.163:0/108899332 wait complete. 2024-07-26T06:18:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.037+0000 7fb1ca600640 1 Processor -- start 2024-07-26T06:18:09.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.037+0000 7fb1ca600640 1 -- start start 2024-07-26T06:18:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.038+0000 7fb1ca600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c413b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:09.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.038+0000 7fb1ca600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40cdd90 0x7fb1c41f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.038+0000 7fb1ca600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c41413b0 0x7fb1c41565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.038+0000 7fb1ca600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c413e860 con 0x7fb1c4074020 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.038+0000 7fb1ca600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c413e6e0 con 0x7fb1c41413b0 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.038+0000 7fb1ca600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c413e9e0 con 0x7fb1c40cdd90 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c413b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c413b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43348/0 (socket says 172.21.15.163:43348) 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 -- 172.21.15.163:0/42551758 learned_addr learned my addr 172.21.15.163:0/42551758 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c2a00640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40cdd90 0x7fb1c41f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 -- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40cdd90 msgr2=0x7fb1c41f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40cdd90 0x7fb1c41f2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 -- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c41413b0 msgr2=0x7fb1c41565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:09.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c41413b0 0x7fb1c41565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 -- 172.21.15.163:0/42551758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c41f2970 con 0x7fb1c4074020 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.039+0000 7fb1c3400640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c413b700 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c7a0 tx=0x7fb1bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.040+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc017070 con 0x7fb1c4074020 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.040+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1bc005070 con 0x7fb1c4074020 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.040+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc005500 con 0x7fb1c4074020 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.040+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c41f2c50 con 0x7fb1c4074020 2024-07-26T06:18:09.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.040+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c41560b0 con 0x7fb1c4074020 2024-07-26T06:18:09.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.041+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c406c810 con 0x7fb1c4074020 2024-07-26T06:18:09.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.041+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 49513+0+0 (secure 0 0 0) 0x7fb1bc01a070 con 0x7fb1c4074020 2024-07-26T06:18:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.042+0000 7fb1bb400640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb1a403d4e0 0x7fb1a403f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.042+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6354+0+0 (secure 0 0 0) 0x7fb1bc051a10 con 0x7fb1c4074020 2024-07-26T06:18:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.042+0000 7fb1c2a00640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb1a403d4e0 0x7fb1a403f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:09.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.043+0000 7fb1c2a00640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb1a403d4e0 0x7fb1a403f9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40b0d90 tx=0x7fb1b0008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:09.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.046+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1c406c810 con 0x7fb1c4074020 2024-07-26T06:18:09.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.274+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb1c413dee0 con 0x7fb1c4074020 2024-07-26T06:18:09.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.495+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb1bc01b3b0 con 0x7fb1c4074020 2024-07-26T06:18:09.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.845+0000 7fb1bb400640 1 -- 172.21.15.163:0/42551758 <== mon.0 v2:172.21.15.163:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) ==== 104+0+0 (secure 0 0 0) 0x7fb1bc064b70 con 0x7fb1c4074020 2024-07-26T06:18:09.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.849+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb1a403d4e0 msgr2=0x7fb1a403f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:09.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.849+0000 7fb1ca600640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb1a403d4e0 0x7fb1a403f9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40b0d90 tx=0x7fb1b0008040 comp rx=0 tx=0).stop 2024-07-26T06:18:09.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 msgr2=0x7fb1c413b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:09.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c413b700 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c7a0 tx=0x7fb1bc00cc60 comp rx=0 tx=0).stop 2024-07-26T06:18:09.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 shutdown_connections 2024-07-26T06:18:09.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb1a403d4e0 0x7fb1a403f9a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c41413b0 0x7fb1c41565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40cdd90 0x7fb1c41f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 --2- 172.21.15.163:0/42551758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb1c4074020 0x7fb1c413b700 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:09.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 >> 172.21.15.163:0/42551758 conn(0x7fb1c406bf40 msgr2=0x7fb1c415d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:09.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 shutdown_connections 2024-07-26T06:18:09.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:09.850+0000 7fb1ca600640 1 -- 172.21.15.163:0/42551758 wait complete. 2024-07-26T06:18:09.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:18:10.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1b200640 1 -- 172.21.15.163:0/1305431024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cfc005180 msgr2=0x7f6cfc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:10.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1b200640 1 --2- 172.21.15.163:0/1305431024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cfc005180 0x7f6cfc00b9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6d14002230 tx=0x7f6cfc005830 comp rx=0 tx=0).stop 2024-07-26T06:18:10.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1a800640 1 -- 172.21.15.163:0/1305431024 reap_dead start 2024-07-26T06:18:10.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1b200640 1 -- 172.21.15.163:0/1305431024 shutdown_connections 2024-07-26T06:18:10.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1b200640 1 -- 172.21.15.163:0/1305431024 >> 172.21.15.163:0/1305431024 conn(0x7f6d1c0b0e30 msgr2=0x7f6d1c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:10.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1b200640 1 -- 172.21.15.163:0/1305431024 shutdown_connections 2024-07-26T06:18:10.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.127+0000 7f6d1b200640 1 -- 172.21.15.163:0/1305431024 wait complete. 2024-07-26T06:18:10.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.129+0000 7f6d1b200640 1 Processor -- start 2024-07-26T06:18:10.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.129+0000 7f6d1b200640 1 -- start start 2024-07-26T06:18:10.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d1b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cfc005180 0x7f6d1c14c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:10.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d1b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 0x7f6d1c153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d1b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d1c0b3420 0x7f6d1c0b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d1b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1c15d560 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d1b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1c06f500 con 0x7f6d1c0b3420 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d1b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6d1c15d6e0 con 0x7f6cfc005180 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d13e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 0x7f6d1c153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d13e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 0x7f6d1c153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43362/0 (socket says 172.21.15.163:43362) 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.130+0000 7f6d13e00640 1 -- 172.21.15.163:0/4255671587 learned_addr learned my addr 172.21.15.163:0/4255671587 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d13e00640 1 -- 172.21.15.163:0/4255671587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cfc005180 msgr2=0x7f6d1c14c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d13e00640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cfc005180 0x7f6d1c14c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d13e00640 1 -- 172.21.15.163:0/4255671587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d1c0b3420 msgr2=0x7f6d1c0b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d13e00640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d1c0b3420 0x7f6d1c0b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d13e00640 1 -- 172.21.15.163:0/4255671587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1c14ce50 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d13e00640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 0x7f6d1c153290 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6d04007860 tx=0x7f6d04007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:10.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04017020 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d0400da30 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04013400 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1c14f930 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.131+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d1c14fef0 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.132+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d1c0cbad0 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.133+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f6d0401a020 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.134+0000 7f6d12000640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cf80763f0 0x7f6cf80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:10.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.134+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6354+0+0 (secure 0 0 0) 0x7f6d04098080 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.135+0000 7f6d19e00640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cf80763f0 0x7f6cf80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:10.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.135+0000 7f6d19e00640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cf80763f0 0x7f6cf80788b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c005fd0 tx=0x7f6d0c005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:10.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.139+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d04021020 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.331+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6d1c0746b0 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.856+0000 7f6d12000640 1 -- 172.21.15.163:0/4255671587 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) ==== 181+0+0 (secure 0 0 0) 0x7f6d04061490 con 0x7f6d1c14f4a0 2024-07-26T06:18:10.858 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:18:10.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cf80763f0 msgr2=0x7f6cf80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cf80763f0 0x7f6cf80788b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c005fd0 tx=0x7f6d0c005850 comp rx=0 tx=0).stop 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 msgr2=0x7f6d1c153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 0x7f6d1c153290 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6d04007860 tx=0x7f6d04007d20 comp rx=0 tx=0).stop 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 shutdown_connections 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cf80763f0 0x7f6cf80788b0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c005fd0 tx=0x7f6d0c005850 comp rx=0 tx=0).stop 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d1c0b3420 0x7f6d1c0b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d1c14f4a0 0x7f6d1c153290 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 --2- 172.21.15.163:0/4255671587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cfc005180 0x7f6d1c14c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.861+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 >> 172.21.15.163:0/4255671587 conn(0x7f6d1c0b0e30 msgr2=0x7f6d1c1e7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:10.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.862+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 shutdown_connections 2024-07-26T06:18:10.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:10.863+0000 7f6d1b200640 1 -- 172.21.15.163:0/4255671587 wait complete. 2024-07-26T06:18:10.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:18:11.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 -- 172.21.15.163:0/3929742051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4558071c40 msgr2=0x7f4558165f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:11.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 --2- 172.21.15.163:0/3929742051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4558071c40 0x7f4558165f90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4550005fd0 tx=0x7f455002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 -- 172.21.15.163:0/3929742051 shutdown_connections 2024-07-26T06:18:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 --2- 172.21.15.163:0/3929742051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4558071c40 0x7f4558165f90 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 -- 172.21.15.163:0/3929742051 >> 172.21.15.163:0/3929742051 conn(0x7f45580b0e30 msgr2=0x7f45580b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 -- 172.21.15.163:0/3929742051 shutdown_connections 2024-07-26T06:18:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.080+0000 7f455da00640 1 -- 172.21.15.163:0/3929742051 wait complete. 2024-07-26T06:18:11.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.082+0000 7f455da00640 1 Processor -- start 2024-07-26T06:18:11.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.082+0000 7f455da00640 1 -- start start 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455da00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4558071c40 0x7f4558158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455da00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45580ce4a0 0x7f4558151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455da00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 0x7f455814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455da00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4558166860 con 0x7f45581556f0 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455da00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45581666e0 con 0x7f4558071c40 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455da00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45581669e0 con 0x7f45580ce4a0 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 0x7f455814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:11.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 0x7f455814f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43366/0 (socket says 172.21.15.163:43366) 2024-07-26T06:18:11.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 -- 172.21.15.163:0/644220280 learned_addr learned my addr 172.21.15.163:0/644220280 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:11.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f454f400640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45580ce4a0 0x7f4558151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:11.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f4557e00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4558071c40 0x7f4558158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 -- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45580ce4a0 msgr2=0x7f4558151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45580ce4a0 0x7f4558151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 -- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4558071c40 msgr2=0x7f4558158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4558071c40 0x7f4558158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.083+0000 7f455d000640 1 -- 172.21.15.163:0/644220280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45581533e0 con 0x7f45581556f0 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.084+0000 7f455d000640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 0x7f455814f360 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f4550006010 tx=0x7f4550003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:11.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.084+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550040070 con 0x7f45581556f0 2024-07-26T06:18:11.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.084+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4550004330 con 0x7f45581556f0 2024-07-26T06:18:11.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.084+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4558159340 con 0x7f45581556f0 2024-07-26T06:18:11.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.084+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550031640 con 0x7f45581556f0 2024-07-26T06:18:11.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.084+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455814f8a0 con 0x7f45581556f0 2024-07-26T06:18:11.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.086+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4558071530 con 0x7f45581556f0 2024-07-26T06:18:11.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.087+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f4550039040 con 0x7f45581556f0 2024-07-26T06:18:11.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.087+0000 7f4554c00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45340763f0 0x7f45340788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:11.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.088+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5916+0+0 (secure 0 0 0) 0x7f45500b9270 con 0x7f45581556f0 2024-07-26T06:18:11.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.088+0000 7f4557e00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45340763f0 0x7f45340788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:11.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.088+0000 7f4557e00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45340763f0 0x7f45340788b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4548006fd0 tx=0x7f4548008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:11.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.092+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f45500828d0 con 0x7f45581556f0 2024-07-26T06:18:11.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.305+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f45580b34f0 con 0x7f45581556f0 2024-07-26T06:18:11.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.868+0000 7f4554c00640 1 -- 172.21.15.163:0/644220280 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) ==== 169+0+0 (secure 0 0 0) 0x7f45500821d0 con 0x7f45581556f0 2024-07-26T06:18:11.870 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:18:11.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45340763f0 msgr2=0x7f45340788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:11.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45340763f0 0x7f45340788b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4548006fd0 tx=0x7f4548008040 comp rx=0 tx=0).stop 2024-07-26T06:18:11.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 msgr2=0x7f455814f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 0x7f455814f360 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f4550006010 tx=0x7f4550003ba0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 shutdown_connections 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45340763f0 0x7f45340788b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45581556f0 0x7f455814f360 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45580ce4a0 0x7f4558151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 --2- 172.21.15.163:0/644220280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4558071c40 0x7f4558158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:11.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 >> 172.21.15.163:0/644220280 conn(0x7f45580b0e30 msgr2=0x7f455814fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:11.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 shutdown_connections 2024-07-26T06:18:11.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:11.872+0000 7f455da00640 1 -- 172.21.15.163:0/644220280 wait complete. 2024-07-26T06:18:11.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-07-26T06:18:12.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 -- 172.21.15.163:0/2530782999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1154071a70 msgr2=0x7f11540b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 --2- 172.21.15.163:0/2530782999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1154071a70 0x7f11540b3710 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f115406cf00 tx=0x7f114c02a8b0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 -- 172.21.15.163:0/2530782999 shutdown_connections 2024-07-26T06:18:12.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 --2- 172.21.15.163:0/2530782999 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1154071a70 0x7f11540b3710 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 --2- 172.21.15.163:0/2530782999 >> v2:172.21.15.164:3300/0 conn(0x7f11540ce4a0 0x7f1154071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 -- 172.21.15.163:0/2530782999 >> 172.21.15.163:0/2530782999 conn(0x7f11540b0e30 msgr2=0x7f11540b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:12.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 -- 172.21.15.163:0/2530782999 shutdown_connections 2024-07-26T06:18:12.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.094+0000 7f115ae00640 1 -- 172.21.15.163:0/2530782999 wait complete. 2024-07-26T06:18:12.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.096+0000 7f115ae00640 1 Processor -- start 2024-07-26T06:18:12.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.096+0000 7f115ae00640 1 -- start start 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f115ae00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1154071a70 0x7f1154154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f115ae00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 0x7f11541469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f115ae00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11541556f0 0x7f1154159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f115ae00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11540b3fe0 con 0x7f11540ce4a0 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f115ae00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f11540b3e60 con 0x7f1154071a70 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f115ae00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f11540b4160 con 0x7f11541556f0 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f1159000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 0x7f11541469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f1159000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 0x7f11541469e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43376/0 (socket says 172.21.15.163:43376) 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.097+0000 7f1159000640 1 -- 172.21.15.163:0/2358563018 learned_addr learned my addr 172.21.15.163:0/2358563018 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159a00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1154071a70 0x7f1154154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f115a400640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11541556f0 0x7f1154159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159000640 1 -- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11541556f0 msgr2=0x7f1154159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159000640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11541556f0 0x7f1154159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159000640 1 -- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1154071a70 msgr2=0x7f1154154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159000640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1154071a70 0x7f1154154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159000640 1 -- 172.21.15.163:0/2358563018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1154146f20 con 0x7f11540ce4a0 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f115a400640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11541556f0 0x7f1154159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1159000640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 0x7f11541469e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f114400b4a0 tx=0x7f114400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.098+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144014070 con 0x7f11540ce4a0 2024-07-26T06:18:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.099+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1144010700 con 0x7f11540ce4a0 2024-07-26T06:18:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.099+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144010b50 con 0x7f11540ce4a0 2024-07-26T06:18:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.099+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115415ae80 con 0x7f11540ce4a0 2024-07-26T06:18:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.099+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115415aba0 con 0x7f11540ce4a0 2024-07-26T06:18:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.100+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11540b4710 con 0x7f11540ce4a0 2024-07-26T06:18:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.101+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f1144017070 con 0x7f11540ce4a0 2024-07-26T06:18:12.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.103+0000 7f1151600640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f11380763f0 0x7f11380788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.103+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7f1144097250 con 0x7f11540ce4a0 2024-07-26T06:18:12.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.103+0000 7f1159a00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f11380763f0 0x7f11380788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.104+0000 7f1159a00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f11380763f0 0x7f11380788b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1148007fd0 tx=0x7f1148009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:12.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.107+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1154146f20 con 0x7f11540ce4a0 2024-07-26T06:18:12.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.315+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f11540b34f0 con 0x7f11540ce4a0 2024-07-26T06:18:12.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.322+0000 7f1151600640 1 -- 172.21.15.163:0/2358563018 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v35) ==== 71+0+0 (secure 0 0 0) 0x7f1144060c00 con 0x7f11540ce4a0 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.325+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f11380763f0 msgr2=0x7f11380788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.325+0000 7f115ae00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f11380763f0 0x7f11380788b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1148007fd0 tx=0x7f1148009040 comp rx=0 tx=0).stop 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.325+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 msgr2=0x7f11541469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.325+0000 7f115ae00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 0x7f11541469e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f114400b4a0 tx=0x7f114400b960 comp rx=0 tx=0).stop 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 shutdown_connections 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f11380763f0 0x7f11380788b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11541556f0 0x7f1154159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f11540ce4a0 0x7f11541469e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 --2- 172.21.15.163:0/2358563018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1154071a70 0x7f1154154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 >> 172.21.15.163:0/2358563018 conn(0x7f11540b0e30 msgr2=0x7f11540cdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 shutdown_connections 2024-07-26T06:18:12.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.326+0000 7f115ae00640 1 -- 172.21.15.163:0/2358563018 wait complete. 2024-07-26T06:18:12.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-07-26T06:18:12.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f0085000640 1 -- 172.21.15.163:0/2795757127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 msgr2=0x7f0060003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f0085000640 1 --2- 172.21.15.163:0/2795757127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f0060003e50 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f007000c470 tx=0x7f0060005830 comp rx=0 tx=0).stop 2024-07-26T06:18:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f007e800640 1 -- 172.21.15.163:0/2795757127 reap_dead start 2024-07-26T06:18:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f0085000640 1 -- 172.21.15.163:0/2795757127 shutdown_connections 2024-07-26T06:18:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f0085000640 1 -- 172.21.15.163:0/2795757127 >> 172.21.15.163:0/2795757127 conn(0x7f00800b1f50 msgr2=0x7f00800b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f0085000640 1 -- 172.21.15.163:0/2795757127 shutdown_connections 2024-07-26T06:18:12.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.552+0000 7f0085000640 1 -- 172.21.15.163:0/2795757127 wait complete. 2024-07-26T06:18:12.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.554+0000 7f0085000640 1 Processor -- start 2024-07-26T06:18:12.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.554+0000 7f0085000640 1 -- start start 2024-07-26T06:18:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.555+0000 7f0085000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f00800ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.555+0000 7f0085000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00800aef60 0x7f00801457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.555+0000 7f0085000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00801f4740 0x7f0080147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.555+0000 7f0085000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f00800cdf20 con 0x7f0060003a30 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.555+0000 7f0085000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f00800cdda0 con 0x7f00801f4740 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.555+0000 7f0085000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0080072d70 con 0x7f00800aef60 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f00800ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f00800ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43388/0 (socket says 172.21.15.163:43388) 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 -- 172.21.15.163:0/3085199936 learned_addr learned my addr 172.21.15.163:0/3085199936 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007d400640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00800aef60 0x7f00801457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007e800640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00801f4740 0x7f0080147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 -- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00800aef60 msgr2=0x7f00801457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00800aef60 0x7f00801457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 -- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00801f4740 msgr2=0x7f0080147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00801f4740 0x7f0080147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 -- 172.21.15.163:0/3085199936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00800adec0 con 0x7f0060003a30 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007e800640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00801f4740 0x7f0080147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.556+0000 7f007de00640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f00800ad8e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f007000c590 tx=0x7f0070006310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.557+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070039070 con 0x7f0060003a30 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.557+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f007002b910 con 0x7f0060003a30 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.557+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00800ae1a0 con 0x7f0060003a30 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.557+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007002bd20 con 0x7f0060003a30 2024-07-26T06:18:12.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.557+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00800ae760 con 0x7f0060003a30 2024-07-26T06:18:12.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.558+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00800af340 con 0x7f0060003a30 2024-07-26T06:18:12.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.559+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f00700035f0 con 0x7f0060003a30 2024-07-26T06:18:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.560+0000 7f0076000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f005c0763f0 0x7f005c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.560+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7f00700b75a0 con 0x7f0060003a30 2024-07-26T06:18:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.560+0000 7f007d400640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f005c0763f0 0x7f005c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.561+0000 7f007d400640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f005c0763f0 0x7f005c0788b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f00800af9f0 tx=0x7f0068006cf0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:12.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.565+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f008015b840 con 0x7f0060003a30 2024-07-26T06:18:12.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.849+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f008006df10 con 0x7f0060003a30 2024-07-26T06:18:12.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.871+0000 7f0076000640 1 -- 172.21.15.163:0/3085199936 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v244) ==== 131+0+0 (secure 0 0 0) 0x7f008006df10 con 0x7f0060003a30 2024-07-26T06:18:12.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f005c0763f0 msgr2=0x7f005c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f005c0763f0 0x7f005c0788b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f00800af9f0 tx=0x7f0068006cf0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 msgr2=0x7f00800ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:12.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f00800ad8e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f007000c590 tx=0x7f0070006310 comp rx=0 tx=0).stop 2024-07-26T06:18:12.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 shutdown_connections 2024-07-26T06:18:12.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f005c0763f0 0x7f005c0788b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00801f4740 0x7f0080147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00800aef60 0x7f00801457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 --2- 172.21.15.163:0/3085199936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0060003a30 0x7f00800ad8e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:12.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 >> 172.21.15.163:0/3085199936 conn(0x7f00800b1f50 msgr2=0x7f00800b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:12.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 shutdown_connections 2024-07-26T06:18:12.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:12.874+0000 7f0085000640 1 -- 172.21.15.163:0/3085199936 wait complete. 2024-07-26T06:18:12.895 INFO:tasks.cephfs_test_runner:test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-07-26T06:18:12.895 INFO:tasks.cephfs_test_runner:Starting test: test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) Test the creating vols, subvols subvolgroups fails when their names uses 2024-07-26T06:18:12.895 INFO:tasks.cephfs_test_runner:test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:18:12.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:13.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 -- 172.21.15.163:0/3551733494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015d540 msgr2=0x7fadd00d2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 --2- 172.21.15.163:0/3551733494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015d540 0x7fadd00d2950 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fadc0005fd0 tx=0x7fadc002a670 comp rx=0 tx=0).stop 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 -- 172.21.15.163:0/3551733494 shutdown_connections 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 --2- 172.21.15.163:0/3551733494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd00d0540 0x7fadd00b6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 --2- 172.21.15.163:0/3551733494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015d540 0x7fadd00d2950 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 -- 172.21.15.163:0/3551733494 >> 172.21.15.163:0/3551733494 conn(0x7fadd00b4fa0 msgr2=0x7fadd00b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 -- 172.21.15.163:0/3551733494 shutdown_connections 2024-07-26T06:18:13.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.088+0000 7fadd7000640 1 -- 172.21.15.163:0/3551733494 wait complete. 2024-07-26T06:18:13.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.090+0000 7fadd7000640 1 Processor -- start 2024-07-26T06:18:13.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.090+0000 7fadd7000640 1 -- start start 2024-07-26T06:18:13.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.091+0000 7fadd7000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd00d0540 0x7fadd0160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.091+0000 7fadd7000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 0x7fadd01582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.091+0000 7fadd7000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015d540 0x7fadd0158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.091+0000 7fadd7000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fadd00b7060 con 0x7fadd00d2e90 2024-07-26T06:18:13.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.091+0000 7fadd7000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fadd00b6ee0 con 0x7fadd00d0540 2024-07-26T06:18:13.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.091+0000 7fadd7000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fadd00b71e0 con 0x7fadd015d540 2024-07-26T06:18:13.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 0x7fadd01582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 0x7fadd01582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43400/0 (socket says 172.21.15.163:43400) 2024-07-26T06:18:13.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 -- 172.21.15.163:0/1183540903 learned_addr learned my addr 172.21.15.163:0/1183540903 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadcfe00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd00d0540 0x7fadd0160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadd5000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015d540 0x7fadd0158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 -- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015d540 msgr2=0x7fadd0158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015d540 0x7fadd0158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 -- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd00d0540 msgr2=0x7fadd0160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd00d0540 0x7fadd0160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.092+0000 7fadc7e00640 1 -- 172.21.15.163:0/1183540903 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd01f7e60 con 0x7fadd00d2e90 2024-07-26T06:18:13.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.093+0000 7fadc7e00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 0x7fadd01582f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fadb800c450 tx=0x7fadb800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:13.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.093+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb80052d0 con 0x7fadd00d2e90 2024-07-26T06:18:13.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.093+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadb8005470 con 0x7fadd00d2e90 2024-07-26T06:18:13.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.093+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8012870 con 0x7fadd00d2e90 2024-07-26T06:18:13.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.093+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd01f8140 con 0x7fadd00d2e90 2024-07-26T06:18:13.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.093+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd01f8700 con 0x7fadd00d2e90 2024-07-26T06:18:13.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.094+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd0159570 con 0x7fadd00d2e90 2024-07-26T06:18:13.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.095+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fadb80035f0 con 0x7fadd00d2e90 2024-07-26T06:18:13.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.096+0000 7fadccc00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fadac0763f0 0x7fadac0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.096+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7fadb80974f0 con 0x7fadd00d2e90 2024-07-26T06:18:13.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.097+0000 7fadcfe00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fadac0763f0 0x7fadac0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.097+0000 7fadcfe00640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fadac0763f0 0x7fadac0788b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fadc0033000 tx=0x7fadc0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:13.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.101+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fadd0159570 con 0x7fadd00d2e90 2024-07-26T06:18:13.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.306+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadd015cf10 con 0x7fadd00d2e90 2024-07-26T06:18:13.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.308+0000 7fadccc00640 1 -- 172.21.15.163:0/1183540903 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+13870 (secure 0 0 0) 0x7fadb8060d70 con 0x7fadd00d2e90 2024-07-26T06:18:13.309 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:13.310 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":73,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:11.858117+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:6833/1648150930":"2024-07-27T06:18:08.838633+0000","172.21.15.164:0/2970427520":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1879372341":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3625624196":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/2212531749":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1000525641":"2024-07-27T06:18:04.414635+0000","172.21.15.164:6816/1554374994":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3906161023":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/83055060":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/3425233229":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1386257824":"2024-07-27T06:18:04.414635+0000","172.21.15.164:6817/1554374994":"2024-07-27T06:18:04.414635+0000","172.21.15.164:0/1931879051":"2024-07-27T06:18:04.414635+0000","172.21.15.163:6835/1648150930":"2024-07-27T06:18:08.838633+0000","172.21.15.164:0/2203391701":"2024-07-27T06:18:04.414635+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:13.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.311+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fadac0763f0 msgr2=0x7fadac0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.311+0000 7fadd7000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fadac0763f0 0x7fadac0788b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fadc0033000 tx=0x7fadc0004730 comp rx=0 tx=0).stop 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.311+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 msgr2=0x7fadd01582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.311+0000 7fadd7000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 0x7fadd01582f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fadb800c450 tx=0x7fadb800c910 comp rx=0 tx=0).stop 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 shutdown_connections 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fadac0763f0 0x7fadac0788b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015d540 0x7fadd0158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadd00d2e90 0x7fadd01582f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 --2- 172.21.15.163:0/1183540903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd00d0540 0x7fadd0160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 >> 172.21.15.163:0/1183540903 conn(0x7fadd00b4fa0 msgr2=0x7fadd00b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 shutdown_connections 2024-07-26T06:18:13.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.312+0000 7fadd7000640 1 -- 172.21.15.163:0/1183540903 wait complete. 2024-07-26T06:18:13.330 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:13:21.677101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:18:13.330 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T06:18:13.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 -- 172.21.15.163:0/1390490405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a40cfc10 msgr2=0x7f08a406bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 --2- 172.21.15.163:0/1390490405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a40cfc10 0x7f08a406bf40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f08a4050980 tx=0x7f089c02a940 comp rx=0 tx=0).stop 2024-07-26T06:18:13.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 -- 172.21.15.163:0/1390490405 shutdown_connections 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 --2- 172.21.15.163:0/1390490405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a40cfc10 0x7f08a406bf40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 --2- 172.21.15.163:0/1390490405 >> v2:172.21.15.164:3300/0 conn(0x7f08a4158e30 0x7f08a40cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 -- 172.21.15.163:0/1390490405 >> 172.21.15.163:0/1390490405 conn(0x7f08a40b1ee0 msgr2=0x7f08a40b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 -- 172.21.15.163:0/1390490405 shutdown_connections 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.525+0000 7f08aa000640 1 -- 172.21.15.163:0/1390490405 wait complete. 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.527+0000 7f08aa000640 1 Processor -- start 2024-07-26T06:18:13.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.527+0000 7f08aa000640 1 -- start start 2024-07-26T06:18:13.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08aa000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08a40708f0 0x7f08a40cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08aa000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 0x7f08a4157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08aa000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08a4158150 0x7f08a414c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08aa000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f08a415c8c0 con 0x7f08a4158e30 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08aa000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f08a406c850 con 0x7f08a40708f0 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08aa000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f08a415ca40 con 0x7f08a4158150 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f089be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 0x7f08a4157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f089be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 0x7f08a4157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43412/0 (socket says 172.21.15.163:43412) 2024-07-26T06:18:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f089be00640 1 -- 172.21.15.163:0/407785653 learned_addr learned my addr 172.21.15.163:0/407785653 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.528+0000 7f08a8c00640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08a40708f0 0x7f08a40cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f089be00640 1 -- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08a4158150 msgr2=0x7f08a414c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f08a9600640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08a4158150 0x7f08a414c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f089be00640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08a4158150 0x7f08a414c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f089be00640 1 -- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08a40708f0 msgr2=0x7f08a40cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f089be00640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08a40708f0 0x7f08a40cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f089be00640 1 -- 172.21.15.163:0/407785653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a414d1c0 con 0x7f08a4158e30 2024-07-26T06:18:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f089be00640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 0x7f08a4157c10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f088c00c450 tx=0x7f088c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:13.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c011790 con 0x7f08a4158e30 2024-07-26T06:18:13.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f088c017070 con 0x7f08a4158e30 2024-07-26T06:18:13.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c011cc0 con 0x7f08a4158e30 2024-07-26T06:18:13.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a414d4a0 con 0x7f08a4158e30 2024-07-26T06:18:13.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.529+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a40b35c0 con 0x7f08a4158e30 2024-07-26T06:18:13.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.531+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a415cf60 con 0x7f08a4158e30 2024-07-26T06:18:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.532+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f088c01c460 con 0x7f08a4158e30 2024-07-26T06:18:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.532+0000 7f08a1600640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08800763f0 0x7f08800788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.533+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7f088c0976a0 con 0x7f08a4158e30 2024-07-26T06:18:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.533+0000 7f08a8c00640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08800763f0 0x7f08800788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.533+0000 7f08a8c00640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08800763f0 0x7f08800788b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0894005fd0 tx=0x7f0894005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:13.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.537+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f088c060f60 con 0x7f08a4158e30 2024-07-26T06:18:13.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.740+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f08a40cc9a0 con 0x7f08a4158e30 2024-07-26T06:18:13.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.741+0000 7f08a1600640 1 -- 172.21.15.163:0/407785653 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f088c060860 con 0x7f08a4158e30 2024-07-26T06:18:13.742 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-07-26T06:18:13.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08800763f0 msgr2=0x7f08800788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08800763f0 0x7f08800788b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0894005fd0 tx=0x7f0894005850 comp rx=0 tx=0).stop 2024-07-26T06:18:13.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 msgr2=0x7f08a4157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 0x7f08a4157c10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f088c00c450 tx=0x7f088c00c910 comp rx=0 tx=0).stop 2024-07-26T06:18:13.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 shutdown_connections 2024-07-26T06:18:13.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08800763f0 0x7f08800788b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08a4158150 0x7f08a414c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08a4158e30 0x7f08a4157c10 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 --2- 172.21.15.163:0/407785653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08a40708f0 0x7f08a40cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 >> 172.21.15.163:0/407785653 conn(0x7f08a40b1ee0 msgr2=0x7f08a40afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:13.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 shutdown_connections 2024-07-26T06:18:13.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.744+0000 7f08aa000640 1 -- 172.21.15.163:0/407785653 wait complete. 2024-07-26T06:18:13.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 -- 172.21.15.163:0/322067225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 msgr2=0x7f03540b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 --2- 172.21.15.163:0/322067225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f03540b64c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f033c005fd0 tx=0x7f033c02a410 comp rx=0 tx=0).stop 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 -- 172.21.15.163:0/322067225 shutdown_connections 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 --2- 172.21.15.163:0/322067225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f03540b64c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 --2- 172.21.15.163:0/322067225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f035415d5a0 0x7f03540ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 -- 172.21.15.163:0/322067225 >> 172.21.15.163:0/322067225 conn(0x7f03540cfda0 msgr2=0x7f03540d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 -- 172.21.15.163:0/322067225 shutdown_connections 2024-07-26T06:18:13.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.969+0000 7f0359200640 1 -- 172.21.15.163:0/322067225 wait complete. 2024-07-26T06:18:13.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.970+0000 7f0359200640 1 Processor -- start 2024-07-26T06:18:13.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.971+0000 7f0359200640 1 -- start start 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.971+0000 7f0359200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03540b1e80 0x7f03540b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.971+0000 7f0359200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f035414e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0359200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f035415d5a0 0x7f03540c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0359200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03540b5130 con 0x7f03540ce830 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0359200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f03540b4fb0 con 0x7f035415d5a0 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0359200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f03540b52b0 con 0x7f03540b1e80 2024-07-26T06:18:13.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f035414e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f035414e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:43426/0 (socket says 172.21.15.163:43426) 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 -- 172.21.15.163:0/1628935150 learned_addr learned my addr 172.21.15.163:0/1628935150 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0353400640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03540b1e80 0x7f03540b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0353e00640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f035415d5a0 0x7f03540c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 -- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03540b1e80 msgr2=0x7f03540b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03540b1e80 0x7f03540b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 -- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f035415d5a0 msgr2=0x7f03540c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f035415d5a0 0x7f03540c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.972+0000 7f0352a00640 1 -- 172.21.15.163:0/1628935150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03540b6520 con 0x7f03540ce830 2024-07-26T06:18:13.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.973+0000 7f0352a00640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f035414e0f0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f033c000c00 tx=0x7f033c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:13.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.973+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c03f070 con 0x7f03540ce830 2024-07-26T06:18:13.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.973+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f033c004890 con 0x7f03540ce830 2024-07-26T06:18:13.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.973+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03540c7660 con 0x7f03540ce830 2024-07-26T06:18:13.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.973+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c035030 con 0x7f03540ce830 2024-07-26T06:18:13.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.973+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f035414ebe0 con 0x7f03540ce830 2024-07-26T06:18:13.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.974+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03540ce2f0 con 0x7f03540ce830 2024-07-26T06:18:13.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.975+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f033c042070 con 0x7f03540ce830 2024-07-26T06:18:13.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.976+0000 7f034b400640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f03300763f0 0x7f03300788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:13.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.977+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7f033c0b8cd0 con 0x7f03540ce830 2024-07-26T06:18:13.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.977+0000 7f0353400640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f03300763f0 0x7f03300788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:13.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.977+0000 7f0353400640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f03300763f0 0x7f03300788b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0344009f20 tx=0x7f03440094f0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:13.979+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f033c0825c0 con 0x7f03540ce830 2024-07-26T06:18:14.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.240+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7f035415b6e0 con 0x7f03540ce830 2024-07-26T06:18:14.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.883+0000 7f034b400640 1 -- 172.21.15.163:0/1628935150 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v246) ==== 149+0+0 (secure 0 0 0) 0x7f033c081ec0 con 0x7f03540ce830 2024-07-26T06:18:14.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.887+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f03300763f0 msgr2=0x7f03300788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.887+0000 7f0359200640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f03300763f0 0x7f03300788b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0344009f20 tx=0x7f03440094f0 comp rx=0 tx=0).stop 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.887+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 msgr2=0x7f035414e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.887+0000 7f0359200640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f035414e0f0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f033c000c00 tx=0x7f033c0039e0 comp rx=0 tx=0).stop 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 shutdown_connections 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f03300763f0 0x7f03300788b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f035415d5a0 0x7f03540c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03540ce830 0x7f035414e0f0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 --2- 172.21.15.163:0/1628935150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03540b1e80 0x7f03540b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 >> 172.21.15.163:0/1628935150 conn(0x7f03540cfda0 msgr2=0x7f0354152c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 shutdown_connections 2024-07-26T06:18:14.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:14.888+0000 7f0359200640 1 -- 172.21.15.163:0/1628935150 wait complete. 2024-07-26T06:18:14.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-07-26T06:18:15.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 -- 172.21.15.163:0/1843556217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 msgr2=0x7f0188165f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 --2- 172.21.15.163:0/1843556217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188165f90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0180005fd0 tx=0x7f018002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 -- 172.21.15.163:0/1843556217 shutdown_connections 2024-07-26T06:18:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 --2- 172.21.15.163:0/1843556217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188165f90 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 -- 172.21.15.163:0/1843556217 >> 172.21.15.163:0/1843556217 conn(0x7f01880b0e30 msgr2=0x7f01880b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 -- 172.21.15.163:0/1843556217 shutdown_connections 2024-07-26T06:18:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.108+0000 7f0187e00640 1 -- 172.21.15.163:0/1843556217 wait complete. 2024-07-26T06:18:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.110+0000 7f0187e00640 1 Processor -- start 2024-07-26T06:18:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.110+0000 7f0187e00640 1 -- start start 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0187e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0187e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01880ce4a0 0x7f0188151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0187e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01881556f0 0x7f018814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0187e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01881666e0 con 0x7f0188071c40 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0187e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0188166860 con 0x7f01880ce4a0 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0187e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01881669e0 con 0x7f01881556f0 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0186a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0186a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40412/0 (socket says 172.21.15.163:40412) 2024-07-26T06:18:15.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.111+0000 7f0186a00640 1 -- 172.21.15.163:0/3859693118 learned_addr learned my addr 172.21.15.163:0/3859693118 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0186a00640 1 -- 172.21.15.163:0/3859693118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01881556f0 msgr2=0x7f018814f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0186a00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01881556f0 0x7f018814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0186a00640 1 -- 172.21.15.163:0/3859693118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01880ce4a0 msgr2=0x7f0188151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0186a00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01880ce4a0 0x7f0188151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0186a00640 1 -- 172.21.15.163:0/3859693118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01881533e0 con 0x7f0188071c40 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0186a00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188158e00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f017800b3f0 tx=0x7f017800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178015020 con 0x7f0188071c40 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0178011920 con 0x7f0188071c40 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178011d70 con 0x7f0188071c40 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.112+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0188159340 con 0x7f0188071c40 2024-07-26T06:18:15.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.113+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018814f8a0 con 0x7f0188071c40 2024-07-26T06:18:15.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.114+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0188071530 con 0x7f0188071c40 2024-07-26T06:18:15.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.115+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f0178018020 con 0x7f0188071c40 2024-07-26T06:18:15.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.116+0000 7f017f400640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f01680763f0 0x7f01680788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.116+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7f0178097560 con 0x7f0188071c40 2024-07-26T06:18:15.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.116+0000 7f0186000640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f01680763f0 0x7f01680788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:15.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.117+0000 7f0186000640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f01680763f0 0x7f01680788b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f01880ce2c0 tx=0x7f01740078d0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:15.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.120+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f017801f030 con 0x7f0188071c40 2024-07-26T06:18:15.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.326+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f01880b34f0 con 0x7f0188071c40 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.327+0000 7f017f400640 1 -- 172.21.15.163:0/3859693118 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v35) ==== 59+0+1998 (secure 0 0 0) 0x7f0178060e70 con 0x7f0188071c40 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:mon advanced mon_op_complaint_time 120 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:mon dev mon_warn_on_pool_no_app false 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:osd advanced osd_op_complaint_time 180.000000 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 5439.226762 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3901.149445 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5414.054049 2024-07-26T06:18:15.328 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4219.540367 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4287.026771 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4403.390021 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3932.844298 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4626.659828 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds 20/20 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds_balancer 20/20 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_ms 1/1 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_frag true 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_scatterstat true 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced mds_op_complaint_time 180.000000 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_verify_scatter true 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced osd_op_complaint_time 180.000000 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_mon_op_timeout 900 2024-07-26T06:18:15.329 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_osd_op_timeout 900 2024-07-26T06:18:15.330 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced client_mount_timeout 600 2024-07-26T06:18:15.330 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_client 20/20 2024-07-26T06:18:15.330 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_ms 1/1 2024-07-26T06:18:15.330 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_mon_op_timeout 900 2024-07-26T06:18:15.330 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_osd_op_timeout 900 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f01680763f0 msgr2=0x7f01680788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f01680763f0 0x7f01680788b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f01880ce2c0 tx=0x7f01740078d0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 msgr2=0x7f0188158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188158e00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f017800b3f0 tx=0x7f017800b8b0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 shutdown_connections 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f01680763f0 0x7f01680788b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01881556f0 0x7f018814f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01880ce4a0 0x7f0188151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 --2- 172.21.15.163:0/3859693118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0188071c40 0x7f0188158e00 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.329+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 >> 172.21.15.163:0/3859693118 conn(0x7f01880b0e30 msgr2=0x7f018814fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.330+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 shutdown_connections 2024-07-26T06:18:15.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.330+0000 7f0187e00640 1 -- 172.21.15.163:0/3859693118 wait complete. 2024-07-26T06:18:15.349 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313e00640 1 -- 172.21.15.163:0/412380885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2f8004d40 msgr2=0x7fb2f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313e00640 1 --2- 172.21.15.163:0/412380885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2f8004d40 0x7fb2f80051c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb3140748f0 tx=0x7fb2f8005830 comp rx=0 tx=0).stop 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313400640 1 -- 172.21.15.163:0/412380885 reap_dead start 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313e00640 1 -- 172.21.15.163:0/412380885 shutdown_connections 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313e00640 1 -- 172.21.15.163:0/412380885 >> 172.21.15.163:0/412380885 conn(0x7fb3140b0e30 msgr2=0x7fb3140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313e00640 1 -- 172.21.15.163:0/412380885 shutdown_connections 2024-07-26T06:18:15.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.552+0000 7fb313e00640 1 -- 172.21.15.163:0/412380885 wait complete. 2024-07-26T06:18:15.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.553+0000 7fb313e00640 1 Processor -- start 2024-07-26T06:18:15.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.554+0000 7fb313e00640 1 -- start start 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.554+0000 7fb313e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2f8004d40 0x7fb314141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.554+0000 7fb313e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 0x7fb31413d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.554+0000 7fb313e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb31413d940 0x7fb3141f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb313e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140cd270 con 0x7fb314158500 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb313e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb31406f500 con 0x7fb31413d940 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb313e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb314159370 con 0x7fb2f8004d40 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 0x7fb31413d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 0x7fb31413d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40444/0 (socket says 172.21.15.163:40444) 2024-07-26T06:18:15.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 -- 172.21.15.163:0/1978787484 learned_addr learned my addr 172.21.15.163:0/1978787484 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:15.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 -- 172.21.15.163:0/1978787484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2f8004d40 msgr2=0x7fb314141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2f8004d40 0x7fb314141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 -- 172.21.15.163:0/1978787484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb31413d940 msgr2=0x7fb3141f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb31413d940 0x7fb3141f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 -- 172.21.15.163:0/1978787484 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31414aad0 con 0x7fb314158500 2024-07-26T06:18:15.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.555+0000 7fb312000640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 0x7fb31413d400 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e970 tx=0x7fb30000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:15.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.556+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30000cc40 con 0x7fb314158500 2024-07-26T06:18:15.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.556+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb30000cde0 con 0x7fb314158500 2024-07-26T06:18:15.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.556+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300005280 con 0x7fb314158500 2024-07-26T06:18:15.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.556+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31414adb0 con 0x7fb314158500 2024-07-26T06:18:15.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.556+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31414d9a0 con 0x7fb314158500 2024-07-26T06:18:15.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.557+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31413dd60 con 0x7fb314158500 2024-07-26T06:18:15.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.558+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb300005420 con 0x7fb314158500 2024-07-26T06:18:15.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.559+0000 7fb30aa00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2f40763f0 0x7fb2f40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:15.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.559+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7fb3000981b0 con 0x7fb314158500 2024-07-26T06:18:15.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.559+0000 7fb312a00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2f40763f0 0x7fb2f40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:15.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.560+0000 7fb312a00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2f40763f0 0x7fb2f40788b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb3040079c0 tx=0x7fb304001700 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:15.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.563+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb3000619b0 con 0x7fb314158500 2024-07-26T06:18:15.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.767+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb314159900 con 0x7fb314158500 2024-07-26T06:18:15.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.773+0000 7fb30aa00640 1 -- 172.21.15.163:0/1978787484 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7fb3000612b0 con 0x7fb314158500 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2f40763f0 msgr2=0x7fb2f40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2f40763f0 0x7fb2f40788b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb3040079c0 tx=0x7fb304001700 comp rx=0 tx=0).stop 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 msgr2=0x7fb31413d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 0x7fb31413d400 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e970 tx=0x7fb30000ee30 comp rx=0 tx=0).stop 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 shutdown_connections 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2f40763f0 0x7fb2f40788b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb31413d940 0x7fb3141f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb314158500 0x7fb31413d400 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 --2- 172.21.15.163:0/1978787484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2f8004d40 0x7fb314141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:15.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 >> 172.21.15.163:0/1978787484 conn(0x7fb3140b0e30 msgr2=0x7fb314152c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:15.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 shutdown_connections 2024-07-26T06:18:15.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:15.776+0000 7fb313e00640 1 -- 172.21.15.163:0/1978787484 wait complete. 2024-07-26T06:18:15.796 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:18:15.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:15.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:16.022 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:18:16.022 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:16.022 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:16.079 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:16.079 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:16.079 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:16.135 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:16.135 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:16.135 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:16.192 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:16.192 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:16.192 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:16.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:16.249 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-07-26T06:18:16.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:16.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 -- 172.21.15.163:0/469715199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b23d0 msgr2=0x7f94d80b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:16.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 --2- 172.21.15.163:0/469715199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b23d0 0x7f94d80b6a00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f94d8160e50 tx=0x7f94d002a440 comp rx=0 tx=0).stop 2024-07-26T06:18:16.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 -- 172.21.15.163:0/469715199 shutdown_connections 2024-07-26T06:18:16.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 --2- 172.21.15.163:0/469715199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b23d0 0x7f94d80b6a00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 --2- 172.21.15.163:0/469715199 >> v2:172.21.15.164:3300/0 conn(0x7f94d80b1e80 0x7f94d80b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 -- 172.21.15.163:0/469715199 >> 172.21.15.163:0/469715199 conn(0x7f94d80cfda0 msgr2=0x7f94d80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:16.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 -- 172.21.15.163:0/469715199 shutdown_connections 2024-07-26T06:18:16.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.460+0000 7f94dd600640 1 -- 172.21.15.163:0/469715199 wait complete. 2024-07-26T06:18:16.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.462+0000 7f94dd600640 1 Processor -- start 2024-07-26T06:18:16.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.462+0000 7f94dd600640 1 -- start start 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94dd600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 0x7f94d814e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94dd600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94d80b23d0 0x7f94d80cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94dd600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94d815d5a0 0x7f94d80b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94dd600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f94d80b4f20 con 0x7f94d80b1e80 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94dd600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f94d80b4da0 con 0x7f94d80b23d0 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94dd600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f94d80b50a0 con 0x7f94d815d5a0 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 0x7f94d814e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 0x7f94d814e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40462/0 (socket says 172.21.15.163:40462) 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.463+0000 7f94d7e00640 1 -- 172.21.15.163:0/1507156944 learned_addr learned my addr 172.21.15.163:0/1507156944 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94cf400640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94d80b23d0 0x7f94d80cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94dcc00640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94d815d5a0 0x7f94d80b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94d7e00640 1 -- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94d815d5a0 msgr2=0x7f94d80b8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:16.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94d7e00640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94d815d5a0 0x7f94d80b8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94d7e00640 1 -- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94d80b23d0 msgr2=0x7f94d80cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:16.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94d7e00640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94d80b23d0 0x7f94d80cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94d7e00640 1 -- 172.21.15.163:0/1507156944 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d80ca3f0 con 0x7f94d80b1e80 2024-07-26T06:18:16.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94cf400640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94d80b23d0 0x7f94d80cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.464+0000 7f94d7e00640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 0x7f94d814e770 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f94c800ba50 tx=0x7f94c800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.465+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8019070 con 0x7f94d80b1e80 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.465+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94c800cd10 con 0x7f94d80b1e80 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.465+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c80146a0 con 0x7f94d80b1e80 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.465+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d814ecb0 con 0x7f94d80b1e80 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.465+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d80ca970 con 0x7f94d80b1e80 2024-07-26T06:18:16.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.466+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d80b5530 con 0x7f94d80b1e80 2024-07-26T06:18:16.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.467+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f94c801c070 con 0x7f94d80b1e80 2024-07-26T06:18:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.469+0000 7f94d4c00640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f94b40763f0 0x7f94b40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.469+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7f94c8097640 con 0x7f94d80b1e80 2024-07-26T06:18:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.469+0000 7f94cf400640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f94b40763f0 0x7f94b40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.470+0000 7f94cf400640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f94b40763f0 0x7f94b40788b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f94d81f70d0 tx=0x7f94c0005e70 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:16.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.473+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94c8060870 con 0x7f94d80b1e80 2024-07-26T06:18:16.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.720+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94d80d0500 con 0x7f94d80b1e80 2024-07-26T06:18:16.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.721+0000 7f94d4c00640 1 -- 172.21.15.163:0/1507156944 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+3920 (secure 0 0 0) 0x7f94c8064b70 con 0x7f94d80b1e80 2024-07-26T06:18:16.722 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:16.722 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":48,"btime":"2024-07-26T06:18:09:842954+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/1969733412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1969733412},{"type":"v1","addr":"172.21.15.163:6834","nonce":1969733412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4191535597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4191535597},{"type":"v1","addr":"172.21.15.164:6836","nonce":4191535597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2909469386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2909469386},{"type":"v1","addr":"172.21.15.164:6837","nonce":2909469386}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6420,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/721975173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":721975173},{"type":"v1","addr":"172.21.15.163:6835","nonce":721975173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[]} 2024-07-26T06:18:16.723 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 48 2024-07-26T06:18:16.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f94b40763f0 msgr2=0x7f94b40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:16.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f94b40763f0 0x7f94b40788b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f94d81f70d0 tx=0x7f94c0005e70 comp rx=0 tx=0).stop 2024-07-26T06:18:16.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 msgr2=0x7f94d814e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:16.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 0x7f94d814e770 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f94c800ba50 tx=0x7f94c800bf10 comp rx=0 tx=0).stop 2024-07-26T06:18:16.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 shutdown_connections 2024-07-26T06:18:16.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f94b40763f0 0x7f94b40788b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f94d815d5a0 0x7f94d80b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94d80b23d0 0x7f94d80cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 --2- 172.21.15.163:0/1507156944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f94d80b1e80 0x7f94d814e770 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:16.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 >> 172.21.15.163:0/1507156944 conn(0x7f94d80cfda0 msgr2=0x7f94d815b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:16.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 shutdown_connections 2024-07-26T06:18:16.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:16.724+0000 7f94dd600640 1 -- 172.21.15.163:0/1507156944 wait complete. 2024-07-26T06:18:16.747 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:18:16.747 INFO:tasks.ceph.mds.a:Stopping old one... 2024-07-26T06:18:16.747 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:18:16.747 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:16.748 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:18:16.748 INFO:tasks.ceph.mds.c:Stopping old one... 2024-07-26T06:18:16.748 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:18:16.748 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:16.748 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:18:16.749 INFO:tasks.ceph.mds.b:Stopping old one... 2024-07-26T06:18:16.749 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:18:16.749 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:16.749 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:18:16.749 INFO:tasks.ceph.mds.d:Stopping old one... 2024-07-26T06:18:16.749 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:18:16.749 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:16.766 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:18:16.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:18:16.769 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:18:16.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:18:16.771 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:18:16.771 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:18:16.781 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:18:16.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:18:16.816 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:18:16.816 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:16.821 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:18:16.821 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:16.827 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:18:16.827 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:16.833 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:18:16.834 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:16.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-07-26T06:18:16.956 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:18:16.957 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:18:16.980 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:18:16.981 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:18:17.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 -- 172.21.15.163:0/1210463889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2880cb7e0 msgr2=0x7fb28815ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:17.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 --2- 172.21.15.163:0/1210463889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2880cb7e0 0x7fb28815ed20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb28806f660 tx=0x7fb27802a4d0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 -- 172.21.15.163:0/1210463889 shutdown_connections 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 --2- 172.21.15.163:0/1210463889 >> v2:172.21.15.164:3300/0 conn(0x7fb288071cf0 0x7fb28815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 --2- 172.21.15.163:0/1210463889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2880cb7e0 0x7fb28815ed20 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 -- 172.21.15.163:0/1210463889 >> 172.21.15.163:0/1210463889 conn(0x7fb28806d420 msgr2=0x7fb28806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 -- 172.21.15.163:0/1210463889 shutdown_connections 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.012+0000 7fb28f000640 1 -- 172.21.15.163:0/1210463889 wait complete. 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.014+0000 7fb28f000640 1 Processor -- start 2024-07-26T06:18:17.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.014+0000 7fb28f000640 1 -- start start 2024-07-26T06:18:17.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28f000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 0x7fb2881f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:17.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28f000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880cb7e0 0x7fb2880ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:17.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28f000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28815c380 0x7fb2880ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:17.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28f000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb288073530 con 0x7fb288071cf0 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28f000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb28806c860 con 0x7fb2880cb7e0 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28f000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb2880736b0 con 0x7fb28815c380 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb287e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 0x7fb2881f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb287e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 0x7fb2881f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40502/0 (socket says 172.21.15.163:40502) 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb287e00640 1 -- 172.21.15.163:0/686477526 learned_addr learned my addr 172.21.15.163:0/686477526 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb27f400640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880cb7e0 0x7fb2880ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.015+0000 7fb28d000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28815c380 0x7fb2880ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb287e00640 1 -- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28815c380 msgr2=0x7fb2880ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb287e00640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28815c380 0x7fb2880ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb287e00640 1 -- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880cb7e0 msgr2=0x7fb2880ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb287e00640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880cb7e0 0x7fb2880ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb287e00640 1 -- 172.21.15.163:0/686477526 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2880a98b0 con 0x7fb288071cf0 2024-07-26T06:18:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb287e00640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 0x7fb2881f36c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb278000c00 tx=0x7fb278003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27803f070 con 0x7fb288071cf0 2024-07-26T06:18:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2780047a0 con 0x7fb288071cf0 2024-07-26T06:18:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2880a9b90 con 0x7fb288071cf0 2024-07-26T06:18:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278035030 con 0x7fb288071cf0 2024-07-26T06:18:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.016+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2880aa150 con 0x7fb288071cf0 2024-07-26T06:18:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.018+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb28815be40 con 0x7fb288071cf0 2024-07-26T06:18:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.019+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb278042070 con 0x7fb288071cf0 2024-07-26T06:18:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.020+0000 7fb284c00640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2640763f0 0x7fb2640788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.020+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5502+0+0 (secure 0 0 0) 0x7fb2780b8650 con 0x7fb288071cf0 2024-07-26T06:18:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.020+0000 7fb27f400640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2640763f0 0x7fb2640788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.021+0000 7fb27f400640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2640763f0 0x7fb2640788b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb2880aabe0 tx=0x7fb27000b390 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:17.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.024+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb278081810 con 0x7fb288071cf0 2024-07-26T06:18:17.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.244+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb28815e9e0 con 0x7fb288071cf0 2024-07-26T06:18:17.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.915+0000 7fb284c00640 1 -- 172.21.15.163:0/686477526 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) ==== 97+0+0 (secure 0 0 0) 0x7fb278085b10 con 0x7fb288071cf0 2024-07-26T06:18:17.917 INFO:teuthology.orchestra.run.smithi163.stderr: removed all blocklist entries 2024-07-26T06:18:17.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2640763f0 msgr2=0x7fb2640788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:17.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2640763f0 0x7fb2640788b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb2880aabe0 tx=0x7fb27000b390 comp rx=0 tx=0).stop 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 msgr2=0x7fb2881f36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 0x7fb2881f36c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb278000c00 tx=0x7fb278003910 comp rx=0 tx=0).stop 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 shutdown_connections 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2640763f0 0x7fb2640788b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28815c380 0x7fb2880ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.919+0000 7fb28f000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880cb7e0 0x7fb2880ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.920+0000 7fb28f000640 1 --2- 172.21.15.163:0/686477526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb288071cf0 0x7fb2881f36c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.920+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 >> 172.21.15.163:0/686477526 conn(0x7fb28806d420 msgr2=0x7fb28806bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.920+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 shutdown_connections 2024-07-26T06:18:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:17.920+0000 7fb28f000640 1 -- 172.21.15.163:0/686477526 wait complete. 2024-07-26T06:18:17.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-07-26T06:18:18.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b63200640 1 -- 172.21.15.163:0/3273066056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b48003a50 msgr2=0x7f6b48003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b63200640 1 --2- 172.21.15.163:0/3273066056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b48003a50 0x7f6b48003e70 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400bb20 tx=0x7f6b48005830 comp rx=0 tx=0).stop 2024-07-26T06:18:18.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b62800640 1 -- 172.21.15.163:0/3273066056 reap_dead start 2024-07-26T06:18:18.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b63200640 1 -- 172.21.15.163:0/3273066056 shutdown_connections 2024-07-26T06:18:18.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b63200640 1 -- 172.21.15.163:0/3273066056 >> 172.21.15.163:0/3273066056 conn(0x7f6b640b0e30 msgr2=0x7f6b640b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:18.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b63200640 1 -- 172.21.15.163:0/3273066056 shutdown_connections 2024-07-26T06:18:18.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.133+0000 7f6b63200640 1 -- 172.21.15.163:0/3273066056 wait complete. 2024-07-26T06:18:18.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.135+0000 7f6b63200640 1 Processor -- start 2024-07-26T06:18:18.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.135+0000 7f6b63200640 1 -- start start 2024-07-26T06:18:18.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b63200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b48003a50 0x7f6b6414bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b63200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 0x7f6b64153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b63200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b641522c0 0x7f6b64154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b63200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64071530 con 0x7f6b6414eeb0 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b63200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64071ee0 con 0x7f6b641522c0 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b63200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6b640716b0 con 0x7f6b48003a50 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 0x7f6b64153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 0x7f6b64153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40520/0 (socket says 172.21.15.163:40520) 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 -- 172.21.15.163:0/640585839 learned_addr learned my addr 172.21.15.163:0/640585839 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61e00640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b48003a50 0x7f6b6414bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 -- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b48003a50 msgr2=0x7f6b6414bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b62800640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b641522c0 0x7f6b64154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b48003a50 0x7f6b6414bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 -- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b641522c0 msgr2=0x7f6b64154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b641522c0 0x7f6b64154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.136+0000 7f6b61400640 1 -- 172.21.15.163:0/640585839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b641540f0 con 0x7f6b6414eeb0 2024-07-26T06:18:18.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.137+0000 7f6b61400640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 0x7f6b64153bb0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000f7f0 tx=0x7f6b5000fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.137+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50015070 con 0x7f6b6414eeb0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.137+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b5000b840 con 0x7f6b6414eeb0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.137+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b64152b10 con 0x7f6b6414eeb0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.137+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5000fea0 con 0x7f6b6414eeb0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.137+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b641530d0 con 0x7f6b6414eeb0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.138+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b640b34f0 con 0x7f6b6414eeb0 2024-07-26T06:18:18.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.139+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f6b5000bc90 con 0x7f6b6414eeb0 2024-07-26T06:18:18.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.141+0000 7f6b5a000640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6b44076420 0x7f6b440788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.141+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f6b50098210 con 0x7f6b6414eeb0 2024-07-26T06:18:18.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.141+0000 7f6b61e00640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6b44076420 0x7f6b440788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.142+0000 7f6b61e00640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6b44076420 0x7f6b440788e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b54036000 tx=0x7f6b540046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:18.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.144+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b50061c00 con 0x7f6b6414eeb0 2024-07-26T06:18:18.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.420+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f6b641e4750 con 0x7f6b6414eeb0 2024-07-26T06:18:18.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.421+0000 7f6b5a000640 1 -- 172.21.15.163:0/640585839 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7f6b50061500 con 0x7f6b6414eeb0 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_dump": [ 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.a", 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA==", 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:18.422 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.b", 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg==", 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.c", 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw==", 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.424 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.d", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg==", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.0", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA==", 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.425 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.1", 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA==", 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.2", 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA==", 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.3", 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg==", 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.427 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.4", 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA==", 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.5", 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g==", 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.6", 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ==", 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.429 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.7", 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA==", 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.0", 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew==", 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.1", 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ==", 2024-07-26T06:18:18.431 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.2", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw==", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:18:18.432 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.3", 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw==", 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.admin", 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm+ibKBxAA1ZBGvVy5rNIzMWwOFwFJFw==", 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.x", 2024-07-26T06:18:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ==", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.y", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ==", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:18.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.436 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:18.436 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6b44076420 msgr2=0x7f6b440788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6b44076420 0x7f6b440788e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b54036000 tx=0x7f6b540046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 msgr2=0x7f6b64153bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 0x7f6b64153bb0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000f7f0 tx=0x7f6b5000fcb0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 shutdown_connections 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6b44076420 0x7f6b440788e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b641522c0 0x7f6b64154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b6414eeb0 0x7f6b64153bb0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 --2- 172.21.15.163:0/640585839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b48003a50 0x7f6b6414bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.423+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 >> 172.21.15.163:0/640585839 conn(0x7f6b640b0e30 msgr2=0x7f6b640b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.424+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 shutdown_connections 2024-07-26T06:18:18.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.424+0000 7f6b63200640 1 -- 172.21.15.163:0/640585839 wait complete. 2024-07-26T06:18:18.441 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:18:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 -- 172.21.15.163:0/1945040355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f01556f0 msgr2=0x7ff9f00cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1945040355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f01556f0 0x7ff9f00cdd90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005fd0 tx=0x7ff9e402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 -- 172.21.15.163:0/1945040355 shutdown_connections 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1945040355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9f00ce2d0 0x7ff9f0071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1945040355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f01556f0 0x7ff9f00cdd90 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 -- 172.21.15.163:0/1945040355 >> 172.21.15.163:0/1945040355 conn(0x7ff9f00b0e30 msgr2=0x7ff9f00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.633+0000 7ff9f6000640 1 -- 172.21.15.163:0/1945040355 shutdown_connections 2024-07-26T06:18:18.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.634+0000 7ff9f6000640 1 -- 172.21.15.163:0/1945040355 wait complete. 2024-07-26T06:18:18.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.635+0000 7ff9f6000640 1 Processor -- start 2024-07-26T06:18:18.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.635+0000 7ff9f6000640 1 -- start start 2024-07-26T06:18:18.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9f6000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9f0071a70 0x7ff9f0157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9f6000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 0x7ff9f0150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9f6000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9f01556f0 0x7ff9f01f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9f6000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f015f310 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9f6000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f0071ee0 con 0x7ff9f01556f0 2024-07-26T06:18:18.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9f6000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f015f490 con 0x7ff9f0071a70 2024-07-26T06:18:18.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9efe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 0x7ff9f0150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9efe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 0x7ff9f0150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40546/0 (socket says 172.21.15.163:40546) 2024-07-26T06:18:18.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.636+0000 7ff9efe00640 1 -- 172.21.15.163:0/1311911006 learned_addr learned my addr 172.21.15.163:0/1311911006 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:18.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9efe00640 1 -- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9f0071a70 msgr2=0x7ff9f0157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9f4c00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9f0071a70 0x7ff9f0157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9efe00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9f0071a70 0x7ff9f0157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9efe00640 1 -- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9f01556f0 msgr2=0x7ff9f01f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9efe00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9f01556f0 0x7ff9f01f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9efe00640 1 -- 172.21.15.163:0/1311911006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f01508b0 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9efe00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 0x7ff9f0150370 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00c970 tx=0x7ff9dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc004220 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9dc0044b0 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc007c10 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.637+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f01578c0 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.638+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f0157e80 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.639+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9f0071530 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.640+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7ff9dc003610 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.641+0000 7ff9ecc00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9d00763f0 0x7ff9d00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:18.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.641+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7ff9dc097ad0 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.641+0000 7ff9f4c00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9d00763f0 0x7ff9d00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:18.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.642+0000 7ff9f4c00640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9d00763f0 0x7ff9d00788b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4033000 tx=0x7ff9e4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:18.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.646+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff9dc014030 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.854+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff9f00b34f0 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.854+0000 7ff9ecc00640 1 -- 172.21.15.163:0/1311911006 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) ==== 84+0+65 (secure 0 0 0) 0x7ff9dc061560 con 0x7ff9f00ce2d0 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:18.856 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:18:18.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.857+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9d00763f0 msgr2=0x7ff9d00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.857+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9d00763f0 0x7ff9d00788b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4033000 tx=0x7ff9e4004730 comp rx=0 tx=0).stop 2024-07-26T06:18:18.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.857+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 msgr2=0x7ff9f0150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:18.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.857+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 0x7ff9f0150370 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00c970 tx=0x7ff9dc00ce30 comp rx=0 tx=0).stop 2024-07-26T06:18:18.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.857+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 shutdown_connections 2024-07-26T06:18:18.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.857+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9d00763f0 0x7ff9d00788b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.858+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9f01556f0 0x7ff9f01f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.858+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9f00ce2d0 0x7ff9f0150370 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.858+0000 7ff9f6000640 1 --2- 172.21.15.163:0/1311911006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9f0071a70 0x7ff9f0157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:18.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.858+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 >> 172.21.15.163:0/1311911006 conn(0x7ff9f00b0e30 msgr2=0x7ff9f00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:18.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.858+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 shutdown_connections 2024-07-26T06:18:18.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:18.858+0000 7ff9f6000640 1 -- 172.21.15.163:0/1311911006 wait complete. 2024-07-26T06:18:18.879 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:18:18.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:18:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 -- 172.21.15.163:0/3060340066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d8071a70 msgr2=0x7fd3d80b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 --2- 172.21.15.163:0/3060340066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d8071a70 0x7fd3d80b3710 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0005fd0 tx=0x7fd3d002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 -- 172.21.15.163:0/3060340066 shutdown_connections 2024-07-26T06:18:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 --2- 172.21.15.163:0/3060340066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d8071a70 0x7fd3d80b3710 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 --2- 172.21.15.163:0/3060340066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d80ce4a0 0x7fd3d8071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 -- 172.21.15.163:0/3060340066 >> 172.21.15.163:0/3060340066 conn(0x7fd3d80b0e30 msgr2=0x7fd3d80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 -- 172.21.15.163:0/3060340066 shutdown_connections 2024-07-26T06:18:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.087+0000 7fd3de400640 1 -- 172.21.15.163:0/3060340066 wait complete. 2024-07-26T06:18:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.089+0000 7fd3de400640 1 Processor -- start 2024-07-26T06:18:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.089+0000 7fd3de400640 1 -- start start 2024-07-26T06:18:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3de400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d8071a70 0x7fd3d81f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3de400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 0x7fd3d8150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3de400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3d81556f0 0x7fd3d80ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3de400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d80b4020 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3de400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d80b3ea0 con 0x7fd3d81556f0 2024-07-26T06:18:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3de400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d80b41a0 con 0x7fd3d8071a70 2024-07-26T06:18:19.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 0x7fd3d8150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:19.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 0x7fd3d8150bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40578/0 (socket says 172.21.15.163:40578) 2024-07-26T06:18:19.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.090+0000 7fd3d7e00640 1 -- 172.21.15.163:0/1232393554 learned_addr learned my addr 172.21.15.163:0/1232393554 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:19.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3d7e00640 1 -- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d8071a70 msgr2=0x7fd3d81f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:19.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3dda00640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3d81556f0 0x7fd3d80ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:19.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3dd000640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d8071a70 0x7fd3d81f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:19.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3d7e00640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d8071a70 0x7fd3d81f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3d7e00640 1 -- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3d81556f0 msgr2=0x7fd3d80ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3d7e00640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3d81556f0 0x7fd3d80ac650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3d7e00640 1 -- 172.21.15.163:0/1232393554 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d80a9eb0 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3dd000640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d8071a70 0x7fd3d81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.091+0000 7fd3d7e00640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 0x7fd3d8150bb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c970 tx=0x7fd3c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.092+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4014070 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.092+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3c4007860 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.092+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4007cb0 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.092+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d80aa150 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.092+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d80aa710 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.093+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d80b4730 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.094+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fd3c4017070 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.095+0000 7fd3d4c00640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd3b8076420 0x7fd3b80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:19.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.095+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7fd3c4097a10 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.095+0000 7fd3dd000640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd3b8076420 0x7fd3b80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:19.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.096+0000 7fd3dd000640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd3b8076420 0x7fd3b80788e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc000f60 tx=0x7fd3cc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:19.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.099+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd3c4061400 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.291+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd3d80ced60 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.942+0000 7fd3d4c00640 1 -- 172.21.15.163:0/1232393554 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) ==== 138+0+0 (secure 0 0 0) 0x7fd3c4060d00 con 0x7fd3d80ce4a0 2024-07-26T06:18:19.943 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:18:19.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd3b8076420 msgr2=0x7fd3b80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:19.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd3b8076420 0x7fd3b80788e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc000f60 tx=0x7fd3cc009040 comp rx=0 tx=0).stop 2024-07-26T06:18:19.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 msgr2=0x7fd3d8150bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:19.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 0x7fd3d8150bb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c970 tx=0x7fd3c400ce30 comp rx=0 tx=0).stop 2024-07-26T06:18:19.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 shutdown_connections 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd3b8076420 0x7fd3b80788e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3d81556f0 0x7fd3d80ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd3d80ce4a0 0x7fd3d8150bb0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 --2- 172.21.15.163:0/1232393554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3d8071a70 0x7fd3d81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 >> 172.21.15.163:0/1232393554 conn(0x7fd3d80b0e30 msgr2=0x7fd3d815d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.946+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 shutdown_connections 2024-07-26T06:18:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:19.947+0000 7fd3de400640 1 -- 172.21.15.163:0/1232393554 wait complete. 2024-07-26T06:18:19.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:18:20.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 -- 172.21.15.163:0/241455006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a415d5a0 msgr2=0x7f75a40ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 --2- 172.21.15.163:0/241455006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a415d5a0 0x7f75a40ce2f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f75a4160e50 tx=0x7f759402a980 comp rx=0 tx=0).stop 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 -- 172.21.15.163:0/241455006 shutdown_connections 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 --2- 172.21.15.163:0/241455006 >> v2:172.21.15.164:3300/0 conn(0x7f75a40ce830 0x7f75a40b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 --2- 172.21.15.163:0/241455006 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a415d5a0 0x7f75a40ce2f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 -- 172.21.15.163:0/241455006 >> 172.21.15.163:0/241455006 conn(0x7f75a40cfda0 msgr2=0x7f75a40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 -- 172.21.15.163:0/241455006 shutdown_connections 2024-07-26T06:18:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.170+0000 7f75a9800640 1 -- 172.21.15.163:0/241455006 wait complete. 2024-07-26T06:18:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.173+0000 7f75a9800640 1 Processor -- start 2024-07-26T06:18:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.173+0000 7f75a9800640 1 -- start start 2024-07-26T06:18:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a9800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 0x7f75a40cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a9800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75a40ce830 0x7f75a4151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a9800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75a415d5a0 0x7f75a41f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a9800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f75a40b5130 con 0x7f75a40b1e80 2024-07-26T06:18:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a9800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f75a40b4fb0 con 0x7f75a40ce830 2024-07-26T06:18:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a9800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f75a40b52b0 con 0x7f75a415d5a0 2024-07-26T06:18:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 0x7f75a40cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 0x7f75a40cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40594/0 (socket says 172.21.15.163:40594) 2024-07-26T06:18:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.174+0000 7f75a3e00640 1 -- 172.21.15.163:0/2347249104 learned_addr learned my addr 172.21.15.163:0/2347249104 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f75a3e00640 1 -- 172.21.15.163:0/2347249104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75a415d5a0 msgr2=0x7f75a41f3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f75a3e00640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75a415d5a0 0x7f75a41f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f75a3e00640 1 -- 172.21.15.163:0/2347249104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75a40ce830 msgr2=0x7f75a4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f75a3e00640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75a40ce830 0x7f75a4151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f75a3e00640 1 -- 172.21.15.163:0/2347249104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a4151b40 con 0x7f75a40b1e80 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f75a3e00640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 0x7f75a40cc780 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7594009bf0 tx=0x7f7594003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594034040 con 0x7f75a40b1e80 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7594004640 con 0x7f75a40b1e80 2024-07-26T06:18:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.175+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594045480 con 0x7f75a40b1e80 2024-07-26T06:18:20.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.176+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a40ccd60 con 0x7f75a40b1e80 2024-07-26T06:18:20.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.176+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a40cd320 con 0x7f75a40b1e80 2024-07-26T06:18:20.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.177+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a40ce2f0 con 0x7f75a40b1e80 2024-07-26T06:18:20.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.179+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f7594037020 con 0x7f75a40b1e80 2024-07-26T06:18:20.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.180+0000 7f759be00640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f75800763f0 0x7f75800788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:20.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.181+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5227+0+0 (secure 0 0 0) 0x7f75940b95d0 con 0x7f75a40b1e80 2024-07-26T06:18:20.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.181+0000 7f75a3400640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f75800763f0 0x7f75800788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:20.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.181+0000 7f75a3400640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f75800763f0 0x7f75800788b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75a415c540 tx=0x7f758c005bf0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:20.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.184+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7594082ff0 con 0x7f75a40b1e80 2024-07-26T06:18:20.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.396+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f75a40d26f0 con 0x7f75a40b1e80 2024-07-26T06:18:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.960+0000 7f759be00640 1 -- 172.21.15.163:0/2347249104 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) ==== 170+0+0 (secure 0 0 0) 0x7f75940828f0 con 0x7f75a40b1e80 2024-07-26T06:18:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.963+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f75800763f0 msgr2=0x7f75800788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.963+0000 7f75a9800640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f75800763f0 0x7f75800788b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75a415c540 tx=0x7f758c005bf0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.963+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 msgr2=0x7f75a40cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.963+0000 7f75a9800640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 0x7f75a40cc780 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7594009bf0 tx=0x7f7594003f90 comp rx=0 tx=0).stop 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.964+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 shutdown_connections 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.964+0000 7f75a9800640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f75800763f0 0x7f75800788b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.964+0000 7f75a9800640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75a415d5a0 0x7f75a41f3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.964+0000 7f75a9800640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75a40ce830 0x7f75a4151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.964+0000 7f75a9800640 1 --2- 172.21.15.163:0/2347249104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f75a40b1e80 0x7f75a40cc780 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.964+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 >> 172.21.15.163:0/2347249104 conn(0x7f75a40cfda0 msgr2=0x7f75a415b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.965+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 shutdown_connections 2024-07-26T06:18:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:20.965+0000 7f75a9800640 1 -- 172.21.15.163:0/2347249104 wait complete. 2024-07-26T06:18:20.997 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.178+0000 7f245ec00640 1 -- 172.21.15.163:0/3353612651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 msgr2=0x7f24580b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.178+0000 7f245ec00640 1 --2- 172.21.15.163:0/3353612651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f24580b2f90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f24580b47b0 tx=0x7f244802a4d0 comp rx=0 tx=0).stop 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.179+0000 7f245ec00640 1 -- 172.21.15.163:0/3353612651 shutdown_connections 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.179+0000 7f245ec00640 1 --2- 172.21.15.163:0/3353612651 >> v2:172.21.15.164:3300/0 conn(0x7f24580cc8e0 0x7f24580ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.179+0000 7f245ec00640 1 --2- 172.21.15.163:0/3353612651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f24580b2f90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.179+0000 7f245ec00640 1 -- 172.21.15.163:0/3353612651 >> 172.21.15.163:0/3353612651 conn(0x7f245806d420 msgr2=0x7f245806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.179+0000 7f245ec00640 1 -- 172.21.15.163:0/3353612651 shutdown_connections 2024-07-26T06:18:21.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.179+0000 7f245ec00640 1 -- 172.21.15.163:0/3353612651 wait complete. 2024-07-26T06:18:21.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.181+0000 7f245ec00640 1 Processor -- start 2024-07-26T06:18:21.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.181+0000 7f245ec00640 1 -- start start 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f245ec00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f2458157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f245ec00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24580cc8e0 0x7f2458158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f245ec00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2458159380 0x7f24580afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f245ec00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24580cd8a0 con 0x7f245806f350 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f245ec00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f24580cd720 con 0x7f24580cc8e0 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f245ec00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f24580cda20 con 0x7f2458159380 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f2457e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f2458157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f2457e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f2458157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40612/0 (socket says 172.21.15.163:40612) 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.182+0000 7f2457e00640 1 -- 172.21.15.163:0/3026617411 learned_addr learned my addr 172.21.15.163:0/3026617411 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f244f400640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24580cc8e0 0x7f2458158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:21.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f245cc00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2458159380 0x7f24580afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2457e00640 1 -- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2458159380 msgr2=0x7f24580afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2457e00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2458159380 0x7f24580afeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2457e00640 1 -- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24580cc8e0 msgr2=0x7f2458158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2457e00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24580cc8e0 0x7f2458158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2457e00640 1 -- 172.21.15.163:0/3026617411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245815a750 con 0x7f245806f350 2024-07-26T06:18:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2457e00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f2458157cd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2448009850 tx=0x7f24480039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:21.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244803f070 con 0x7f245806f350 2024-07-26T06:18:21.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2448004890 con 0x7f245806f350 2024-07-26T06:18:21.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448035030 con 0x7f245806f350 2024-07-26T06:18:21.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.183+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245815a9f0 con 0x7f245806f350 2024-07-26T06:18:21.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.184+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f245815afb0 con 0x7f245806f350 2024-07-26T06:18:21.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.185+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24580b2f90 con 0x7f245806f350 2024-07-26T06:18:21.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.186+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f2448042070 con 0x7f245806f350 2024-07-26T06:18:21.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.187+0000 7f2454c00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f24340763f0 0x7f24340788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:21.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.187+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5609+0+0 (secure 0 0 0) 0x7f24480b8ce0 con 0x7f245806f350 2024-07-26T06:18:21.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.188+0000 7f244f400640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f24340763f0 0x7f24340788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:21.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.188+0000 7f244f400640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f24340763f0 0x7f24340788b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2440006fd0 tx=0x7f2440008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:21.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.191+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24480824f0 con 0x7f245806f350 2024-07-26T06:18:21.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.431+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f24581492a0 con 0x7f245806f350 2024-07-26T06:18:21.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.988+0000 7f2454c00640 1 -- 172.21.15.163:0/3026617411 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v51) ==== 174+0+0 (secure 0 0 0) 0x7f2448081df0 con 0x7f245806f350 2024-07-26T06:18:21.990 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 12 and data pool 13 2024-07-26T06:18:21.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.997+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f24340763f0 msgr2=0x7f24340788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:21.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.997+0000 7f245ec00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f24340763f0 0x7f24340788b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2440006fd0 tx=0x7f2440008040 comp rx=0 tx=0).stop 2024-07-26T06:18:21.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.997+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 msgr2=0x7f2458157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:21.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.997+0000 7f245ec00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f2458157cd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2448009850 tx=0x7f24480039e0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 shutdown_connections 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f24340763f0 0x7f24340788b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2458159380 0x7f24580afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24580cc8e0 0x7f2458158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 --2- 172.21.15.163:0/3026617411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f245806f350 0x7f2458157cd0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 >> 172.21.15.163:0/3026617411 conn(0x7f245806d420 msgr2=0x7f2458154ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 shutdown_connections 2024-07-26T06:18:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:21.999+0000 7f245ec00640 1 -- 172.21.15.163:0/3026617411 wait complete. 2024-07-26T06:18:22.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:22.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 -- 172.21.15.163:0/2250654916 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 msgr2=0x7f87240ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 --2- 172.21.15.163:0/2250654916 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87240ad8e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8714005fd0 tx=0x7f871402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 -- 172.21.15.163:0/2250654916 shutdown_connections 2024-07-26T06:18:22.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 --2- 172.21.15.163:0/2250654916 >> v2:172.21.15.164:3300/0 conn(0x7f87240ade20 0x7f87240cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 --2- 172.21.15.163:0/2250654916 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87240ad8e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 -- 172.21.15.163:0/2250654916 >> 172.21.15.163:0/2250654916 conn(0x7f87240b0e30 msgr2=0x7f87240b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:22.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 -- 172.21.15.163:0/2250654916 shutdown_connections 2024-07-26T06:18:22.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.207+0000 7f8723e00640 1 -- 172.21.15.163:0/2250654916 wait complete. 2024-07-26T06:18:22.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.209+0000 7f8723e00640 1 Processor -- start 2024-07-26T06:18:22.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.209+0000 7f8723e00640 1 -- start start 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8723e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87240ade20 0x7f872413ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8723e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 0x7f87241f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8723e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87241f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8723e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f87240b4970 con 0x7f87240cc280 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8723e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87240cc830 con 0x7f87240ade20 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8723e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87240ce520 con 0x7f8724146be0 2024-07-26T06:18:22.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8722000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 0x7f87241f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8722000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 0x7f87241f39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40616/0 (socket says 172.21.15.163:40616) 2024-07-26T06:18:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.210+0000 7f8722000640 1 -- 172.21.15.163:0/3002101021 learned_addr learned my addr 172.21.15.163:0/3002101021 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722a00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87240ade20 0x7f872413ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722000640 1 -- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 msgr2=0x7f87241f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8723400640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87241f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722000640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87241f3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722000640 1 -- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87240ade20 msgr2=0x7f872413ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722000640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87240ade20 0x7f872413ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722000640 1 -- 172.21.15.163:0/3002101021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872413f410 con 0x7f87240cc280 2024-07-26T06:18:22.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8723400640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87241f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.211+0000 7f8722000640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 0x7f87241f39c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f871000ea60 tx=0x7f871000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.212+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871000cde0 con 0x7f87240cc280 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.212+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8710004020 con 0x7f87240cc280 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.212+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710004450 con 0x7f87240cc280 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.212+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872413f6f0 con 0x7f87240cc280 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.212+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872413fcb0 con 0x7f87240cc280 2024-07-26T06:18:22.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.213+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87240cea80 con 0x7f87240cc280 2024-07-26T06:18:22.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.214+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f87100035f0 con 0x7f87240cc280 2024-07-26T06:18:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.215+0000 7f871aa00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f87040763f0 0x7f87040788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.215+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f8710098470 con 0x7f87240cc280 2024-07-26T06:18:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.216+0000 7f8722a00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f87040763f0 0x7f87040788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.216+0000 7f8722a00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f87040763f0 0x7f87040788b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8714033000 tx=0x7f8714004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:22.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.219+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8710061a90 con 0x7f87240cc280 2024-07-26T06:18:22.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.431+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87240af080 con 0x7f87240cc280 2024-07-26T06:18:22.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.433+0000 7f871aa00640 1 -- 172.21.15.163:0/3002101021 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16697 (secure 0 0 0) 0x7f8710061390 con 0x7f87240cc280 2024-07-26T06:18:22.435 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:22.435 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":80,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:21.433870+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:19.305919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:20.414373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f87040763f0 msgr2=0x7f87040788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f87040763f0 0x7f87040788b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8714033000 tx=0x7f8714004730 comp rx=0 tx=0).stop 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 msgr2=0x7f87241f39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 0x7f87241f39c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f871000ea60 tx=0x7f871000ef20 comp rx=0 tx=0).stop 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 shutdown_connections 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f87040763f0 0x7f87040788b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724146be0 0x7f87241f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87240cc280 0x7f87241f39c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 --2- 172.21.15.163:0/3002101021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87240ade20 0x7f872413ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 >> 172.21.15.163:0/3002101021 conn(0x7f87240b0e30 msgr2=0x7f872415f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:22.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 shutdown_connections 2024-07-26T06:18:22.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.437+0000 7f8723e00640 1 -- 172.21.15.163:0/3002101021 wait complete. 2024-07-26T06:18:22.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:22.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 -- 172.21.15.163:0/3274448163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb728005910 msgr2=0x7fb728005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 --2- 172.21.15.163:0/3274448163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb728005910 0x7fb728005d90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009b20 tx=0x7fb728006940 comp rx=0 tx=0).stop 2024-07-26T06:18:22.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 -- 172.21.15.163:0/3274448163 shutdown_connections 2024-07-26T06:18:22.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 --2- 172.21.15.163:0/3274448163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb728005910 0x7fb728005d90 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 -- 172.21.15.163:0/3274448163 >> 172.21.15.163:0/3274448163 conn(0x7fb7400b0e30 msgr2=0x7fb7400b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:22.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 -- 172.21.15.163:0/3274448163 shutdown_connections 2024-07-26T06:18:22.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.663+0000 7fb746800640 1 -- 172.21.15.163:0/3274448163 wait complete. 2024-07-26T06:18:22.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.665+0000 7fb746800640 1 Processor -- start 2024-07-26T06:18:22.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.666+0000 7fb746800640 1 -- start start 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb746800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb728005910 0x7fb7400741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb746800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb740144a50 0x7fb7400b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb746800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 0x7fb7400723d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb746800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb7400cd530 con 0x7fb7400b2670 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb746800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb74006f500 con 0x7fb740144a50 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb746800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb7400cd6b0 con 0x7fb728005910 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb745e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 0x7fb7400723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb745e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 0x7fb7400723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40632/0 (socket says 172.21.15.163:40632) 2024-07-26T06:18:22.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb745e00640 1 -- 172.21.15.163:0/4086427596 learned_addr learned my addr 172.21.15.163:0/4086427596 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:22.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb745400640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb728005910 0x7fb7400741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.667+0000 7fb744a00640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb740144a50 0x7fb7400b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745e00640 1 -- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb728005910 msgr2=0x7fb7400741e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745e00640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb728005910 0x7fb7400741e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745e00640 1 -- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb740144a50 msgr2=0x7fb7400b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745e00640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb740144a50 0x7fb7400b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745e00640 1 -- 172.21.15.163:0/4086427596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb740073140 con 0x7fb7400b2670 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745400640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb728005910 0x7fb7400741e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb745e00640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 0x7fb7400723d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb7400b4340 tx=0x7fb73800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb738018070 con 0x7fb7400b2670 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb738003ca0 con 0x7fb7400b2670 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb738005330 con 0x7fb7400b2670 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.668+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb740073420 con 0x7fb7400b2670 2024-07-26T06:18:22.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.669+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7400739e0 con 0x7fb7400b2670 2024-07-26T06:18:22.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.670+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb740159370 con 0x7fb7400b2670 2024-07-26T06:18:22.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.671+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb73801b070 con 0x7fb7400b2670 2024-07-26T06:18:22.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.672+0000 7fb73d600640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb7200763f0 0x7fb7200788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:22.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.672+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fb738096bb0 con 0x7fb7400b2670 2024-07-26T06:18:22.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.672+0000 7fb745400640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb7200763f0 0x7fb7200788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:22.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.673+0000 7fb745400640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb7200763f0 0x7fb7200788b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb74006b090 tx=0x7fb734009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:22.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.677+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb73805fd10 con 0x7fb7400b2670 2024-07-26T06:18:22.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.885+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7401588d0 con 0x7fb7400b2670 2024-07-26T06:18:22.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.887+0000 7fb73d600640 1 -- 172.21.15.163:0/4086427596 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16697 (secure 0 0 0) 0x7fb738068350 con 0x7fb7400b2670 2024-07-26T06:18:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:22.889 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":80,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:21.433870+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:19.305919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:20.414373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb7200763f0 msgr2=0x7fb7200788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb7200763f0 0x7fb7200788b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb74006b090 tx=0x7fb734009040 comp rx=0 tx=0).stop 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 msgr2=0x7fb7400723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 0x7fb7400723d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb7400b4340 tx=0x7fb73800c960 comp rx=0 tx=0).stop 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 shutdown_connections 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb7200763f0 0x7fb7200788b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7400b2670 0x7fb7400723d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb740144a50 0x7fb7400b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 --2- 172.21.15.163:0/4086427596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb728005910 0x7fb7400741e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:22.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 >> 172.21.15.163:0/4086427596 conn(0x7fb7400b0e30 msgr2=0x7fb74006d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:22.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 shutdown_connections 2024-07-26T06:18:22.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:22.890+0000 7fb746800640 1 -- 172.21.15.163:0/4086427596 wait complete. 2024-07-26T06:18:22.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:18:23.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.123+0000 7fbb55a00640 1 -- 172.21.15.163:0/4256881764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 msgr2=0x7fbb5015ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.123+0000 7fbb55a00640 1 --2- 172.21.15.163:0/4256881764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb5015ed20 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbb40005fd0 tx=0x7fbb4002a540 comp rx=0 tx=0).stop 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.123+0000 7fbb55a00640 1 -- 172.21.15.163:0/4256881764 shutdown_connections 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.123+0000 7fbb55a00640 1 --2- 172.21.15.163:0/4256881764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb50071cf0 0x7fbb5015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.123+0000 7fbb55a00640 1 --2- 172.21.15.163:0/4256881764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb5015ed20 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.123+0000 7fbb55a00640 1 -- 172.21.15.163:0/4256881764 >> 172.21.15.163:0/4256881764 conn(0x7fbb5006d420 msgr2=0x7fbb5006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.124+0000 7fbb55a00640 1 -- 172.21.15.163:0/4256881764 shutdown_connections 2024-07-26T06:18:23.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.124+0000 7fbb55a00640 1 -- 172.21.15.163:0/4256881764 wait complete. 2024-07-26T06:18:23.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.125+0000 7fbb55a00640 1 Processor -- start 2024-07-26T06:18:23.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.125+0000 7fbb55a00640 1 -- start start 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb55a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb50071cf0 0x7fbb501f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb55a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb501f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb55a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5015c380 0x7fbb5014fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb55a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50073530 con 0x7fbb500cb7e0 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb55a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbb500736b0 con 0x7fbb5015c380 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb55a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbb50073830 con 0x7fbb50071cf0 2024-07-26T06:18:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.126+0000 7fbb4e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb501f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:23.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb501f49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40660/0 (socket says 172.21.15.163:40660) 2024-07-26T06:18:23.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 -- 172.21.15.163:0/3889007219 learned_addr learned my addr 172.21.15.163:0/3889007219 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 -- 172.21.15.163:0/3889007219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb50071cf0 msgr2=0x7fbb501f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb50071cf0 0x7fbb501f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 -- 172.21.15.163:0/3889007219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5015c380 msgr2=0x7fbb5014fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5015c380 0x7fbb5014fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 -- 172.21.15.163:0/3889007219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb501f4ee0 con 0x7fbb500cb7e0 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb4e000640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb501f49a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c970 tx=0x7fbb3c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c007bb0 con 0x7fbb500cb7e0 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb3c007e40 con 0x7fbb500cb7e0 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c005280 con 0x7fbb500cb7e0 2024-07-26T06:18:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.127+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb50150390 con 0x7fbb500cb7e0 2024-07-26T06:18:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.128+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb50150ad0 con 0x7fbb500cb7e0 2024-07-26T06:18:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.129+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb50073d50 con 0x7fbb500cb7e0 2024-07-26T06:18:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.130+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fbb3c005420 con 0x7fbb500cb7e0 2024-07-26T06:18:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.131+0000 7fbb46a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbb30076420 0x7fbb300788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:23.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.131+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fbb3c097ed0 con 0x7fbb500cb7e0 2024-07-26T06:18:23.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.131+0000 7fbb4ea00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbb30076420 0x7fbb300788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:23.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.132+0000 7fbb4ea00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbb30076420 0x7fbb300788e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb40033000 tx=0x7fbb400046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:23.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.135+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb3c014030 con 0x7fbb500cb7e0 2024-07-26T06:18:23.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:23.384+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fbb5015ea20 con 0x7fbb500cb7e0 2024-07-26T06:18:23.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2024-07-26T06:18:23.923+0000 7f5535c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:18:24.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.009+0000 7fbb46a00640 1 -- 172.21.15.163:0/3889007219 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) ==== 118+0+0 (secure 0 0 0) 0x7fbb3c061570 con 0x7fbb500cb7e0 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.012+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbb30076420 msgr2=0x7fbb300788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.012+0000 7fbb55a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbb30076420 0x7fbb300788e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb40033000 tx=0x7fbb400046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.012+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 msgr2=0x7fbb501f49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.012+0000 7fbb55a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb501f49a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c970 tx=0x7fbb3c00ce30 comp rx=0 tx=0).stop 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 shutdown_connections 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbb30076420 0x7fbb300788e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5015c380 0x7fbb5014fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb500cb7e0 0x7fbb501f49a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 --2- 172.21.15.163:0/3889007219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb50071cf0 0x7fbb501f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 >> 172.21.15.163:0/3889007219 conn(0x7fbb5006d420 msgr2=0x7fbb500b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:24.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 shutdown_connections 2024-07-26T06:18:24.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.013+0000 7fbb55a00640 1 -- 172.21.15.163:0/3889007219 wait complete. 2024-07-26T06:18:24.032 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:18:24.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:18:24.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.237+0000 7fdc95400640 1 -- 172.21.15.163:0/3417957405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc74004d40 msgr2=0x7fdc740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:24.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.237+0000 7fdc95400640 1 --2- 172.21.15.163:0/3417957405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc74004d40 0x7fdc740051c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009ed0 tx=0x7fdc74005830 comp rx=0 tx=0).stop 2024-07-26T06:18:24.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.237+0000 7fdc8f200640 1 -- 172.21.15.163:0/3417957405 reap_dead start 2024-07-26T06:18:24.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.237+0000 7fdc95400640 1 -- 172.21.15.163:0/3417957405 shutdown_connections 2024-07-26T06:18:24.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.237+0000 7fdc95400640 1 -- 172.21.15.163:0/3417957405 >> 172.21.15.163:0/3417957405 conn(0x7fdc900b4fe0 msgr2=0x7fdc900b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:24.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.238+0000 7fdc95400640 1 -- 172.21.15.163:0/3417957405 shutdown_connections 2024-07-26T06:18:24.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.238+0000 7fdc95400640 1 -- 172.21.15.163:0/3417957405 wait complete. 2024-07-26T06:18:24.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.239+0000 7fdc95400640 1 Processor -- start 2024-07-26T06:18:24.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.239+0000 7fdc95400640 1 -- start start 2024-07-26T06:18:24.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.240+0000 7fdc95400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc74004d40 0x7fdc901f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:24.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.240+0000 7fdc95400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 0x7fdc900b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:24.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.240+0000 7fdc95400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc900b4330 0x7fdc901f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:24.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.240+0000 7fdc95400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900b66a0 con 0x7fdc901f6710 2024-07-26T06:18:24.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.240+0000 7fdc95400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900b6520 con 0x7fdc74004d40 2024-07-26T06:18:24.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.240+0000 7fdc95400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdc900b6820 con 0x7fdc900b4330 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 0x7fdc900b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 0x7fdc900b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:40678/0 (socket says 172.21.15.163:40678) 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 -- 172.21.15.163:0/400851032 learned_addr learned my addr 172.21.15.163:0/400851032 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8e800640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc74004d40 0x7fdc901f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8f200640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc900b4330 0x7fdc901f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 -- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc900b4330 msgr2=0x7fdc901f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc900b4330 0x7fdc901f6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 -- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc74004d40 msgr2=0x7fdc901f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc74004d40 0x7fdc901f4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8de00640 1 -- 172.21.15.163:0/400851032 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc900ab180 con 0x7fdc901f6710 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.241+0000 7fdc8e800640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc74004d40 0x7fdc901f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:24.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.242+0000 7fdc8de00640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 0x7fdc900b3df0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fdc78006fd0 tx=0x7fdc7800dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:24.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.242+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78019070 con 0x7fdc901f6710 2024-07-26T06:18:24.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.242+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc780052e0 con 0x7fdc901f6710 2024-07-26T06:18:24.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.242+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7800b8c0 con 0x7fdc901f6710 2024-07-26T06:18:24.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.242+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc901f7ba0 con 0x7fdc901f6710 2024-07-26T06:18:24.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.242+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc900ab720 con 0x7fdc901f6710 2024-07-26T06:18:24.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.243+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc900ab920 con 0x7fdc901f6710 2024-07-26T06:18:24.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.244+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fdc7800ba60 con 0x7fdc901f6710 2024-07-26T06:18:24.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.245+0000 7fdc86a00640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc6c0763f0 0x7fdc6c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:24.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.245+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fdc780970f0 con 0x7fdc901f6710 2024-07-26T06:18:24.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.246+0000 7fdc8e800640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc6c0763f0 0x7fdc6c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:24.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.246+0000 7fdc8e800640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc6c0763f0 0x7fdc6c0788b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009b10 tx=0x7fdc80005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.250+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc900ab920 con 0x7fdc901f6710 2024-07-26T06:18:24.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:24.502+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdc900d2920 con 0x7fdc901f6710 2024-07-26T06:18:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.012+0000 7fdc86a00640 1 -- 172.21.15.163:0/400851032 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) ==== 122+0+0 (secure 0 0 0) 0x7fdc78060250 con 0x7fdc901f6710 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc6c0763f0 msgr2=0x7fdc6c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc6c0763f0 0x7fdc6c0788b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009b10 tx=0x7fdc80005f90 comp rx=0 tx=0).stop 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 msgr2=0x7fdc900b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 0x7fdc900b3df0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fdc78006fd0 tx=0x7fdc7800dca0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 shutdown_connections 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc6c0763f0 0x7fdc6c0788b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc900b4330 0x7fdc901f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc901f6710 0x7fdc900b3df0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 --2- 172.21.15.163:0/400851032 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc74004d40 0x7fdc901f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 >> 172.21.15.163:0/400851032 conn(0x7fdc900b4fe0 msgr2=0x7fdc9015fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:25.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 shutdown_connections 2024-07-26T06:18:25.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.016+0000 7fdc95400640 1 -- 172.21.15.163:0/400851032 wait complete. 2024-07-26T06:18:25.038 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:18:25.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.240+0000 7f3be7e00640 1 -- 172.21.15.163:0/897517186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc004800 msgr2=0x7f3bcc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.240+0000 7f3be7e00640 1 --2- 172.21.15.163:0/897517186 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bcc004800 0x7f3bcc004c80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000baf0 tx=0x7f3bcc005830 comp rx=0 tx=0).stop 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.241+0000 7f3be7400640 1 -- 172.21.15.163:0/897517186 reap_dead start 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.241+0000 7f3be7e00640 1 -- 172.21.15.163:0/897517186 shutdown_connections 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.241+0000 7f3be7e00640 1 -- 172.21.15.163:0/897517186 >> 172.21.15.163:0/897517186 conn(0x7f3be80b0e30 msgr2=0x7f3be80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.241+0000 7f3be7e00640 1 -- 172.21.15.163:0/897517186 shutdown_connections 2024-07-26T06:18:25.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.241+0000 7f3be7e00640 1 -- 172.21.15.163:0/897517186 wait complete. 2024-07-26T06:18:25.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.242+0000 7f3be7e00640 1 Processor -- start 2024-07-26T06:18:25.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.243+0000 7f3be7e00640 1 -- start start 2024-07-26T06:18:25.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.243+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bcc004800 0x7f3be8155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.243+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be8149970 0x7f3be81e3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.243+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 0x7f3be80ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.243+0000 7f3be7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3be80cda70 con 0x7f3be8149070 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.243+0000 7f3be7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3be806f500 con 0x7f3be8149970 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3be80cdbf0 con 0x7f3bcc004800 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 0x7f3be80ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 0x7f3be80ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55230/0 (socket says 172.21.15.163:55230) 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 -- 172.21.15.163:0/4229421015 learned_addr learned my addr 172.21.15.163:0/4229421015 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3bdfe00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be8149970 0x7f3be81e3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be6a00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bcc004800 0x7f3be8155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 -- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bcc004800 msgr2=0x7f3be8155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bcc004800 0x7f3be8155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 -- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be8149970 msgr2=0x7f3be81e3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be8149970 0x7f3be81e3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3be7400640 1 -- 172.21.15.163:0/4229421015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be815c9d0 con 0x7f3be8149070 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.244+0000 7f3bdfe00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be8149970 0x7f3be81e3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.245+0000 7f3be7400640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 0x7f3be80ad8e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3be80b1e30 tx=0x7f3be000b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.245+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0003970 con 0x7f3be8149070 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.245+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3be0003c00 con 0x7f3be8149070 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.245+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0006070 con 0x7f3be8149070 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.245+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be815bdc0 con 0x7f3be8149070 2024-07-26T06:18:25.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.245+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be815c380 con 0x7f3be8149070 2024-07-26T06:18:25.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.246+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be80728b0 con 0x7f3be8149070 2024-07-26T06:18:25.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.248+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f3be0006210 con 0x7f3be8149070 2024-07-26T06:18:25.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.249+0000 7f3bdea00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3bc40763f0 0x7f3bc40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.249+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f3be0097f70 con 0x7f3be8149070 2024-07-26T06:18:25.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.249+0000 7f3be6a00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3bc40763f0 0x7f3bc40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.250+0000 7f3be6a00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3bc40763f0 0x7f3bc40788b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80098c0 tx=0x7f3bd8005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:25.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.253+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3be00617c0 con 0x7f3be8149070 2024-07-26T06:18:25.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.498+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be80aff10 con 0x7f3be8149070 2024-07-26T06:18:25.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.499+0000 7f3bdea00640 1 -- 172.21.15.163:0/4229421015 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5234 (secure 0 0 0) 0x7f3be00610c0 con 0x7f3be8149070 2024-07-26T06:18:25.499 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:25.499 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":55,"btime":"2024-07-26T06:18:25:009001+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/2494990100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":2494990100},{"type":"v1","addr":"172.21.15.163:6834","nonce":2494990100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:21.433788+0000","modified":"2024-07-26T06:18:24.503309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6516":{"gid":6516,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6834/3348438490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3348438490},{"type":"v1","addr":"172.21.15.164:6834","nonce":3348438490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6516,"qdb_cluster":[6516]},"id":6}]} 2024-07-26T06:18:25.500 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 55 2024-07-26T06:18:25.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3bc40763f0 msgr2=0x7f3bc40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3bc40763f0 0x7f3bc40788b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80098c0 tx=0x7f3bd8005f90 comp rx=0 tx=0).stop 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 msgr2=0x7f3be80ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 0x7f3be80ad8e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3be80b1e30 tx=0x7f3be000b620 comp rx=0 tx=0).stop 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 shutdown_connections 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3bc40763f0 0x7f3bc40788b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3be8149070 0x7f3be80ad8e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be8149970 0x7f3be81e3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.502+0000 7f3be7e00640 1 --2- 172.21.15.163:0/4229421015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bcc004800 0x7f3be8155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.503+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 >> 172.21.15.163:0/4229421015 conn(0x7f3be80b0e30 msgr2=0x7f3be814b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.503+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 shutdown_connections 2024-07-26T06:18:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.503+0000 7f3be7e00640 1 -- 172.21.15.163:0/4229421015 wait complete. 2024-07-26T06:18:25.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 -- 172.21.15.163:0/963841301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0004800 msgr2=0x7fb2b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 --2- 172.21.15.163:0/963841301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0004800 0x7fb2b0004c80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400baf0 tx=0x7fb2b0005830 comp rx=0 tx=0).stop 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 -- 172.21.15.163:0/963841301 shutdown_connections 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 --2- 172.21.15.163:0/963841301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0004800 0x7fb2b0004c80 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 -- 172.21.15.163:0/963841301 >> 172.21.15.163:0/963841301 conn(0x7fb2c80b0e30 msgr2=0x7fb2c80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 -- 172.21.15.163:0/963841301 shutdown_connections 2024-07-26T06:18:25.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.715+0000 7fb2cec00640 1 -- 172.21.15.163:0/963841301 wait complete. 2024-07-26T06:18:25.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.716+0000 7fb2cec00640 1 Processor -- start 2024-07-26T06:18:25.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 -- start start 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2b0004800 0x7fb2c814bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c814a8c0 0x7fb2c8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 0x7fb2c81f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c80718c0 con 0x7fb2c8156150 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8071740 con 0x7fb2c814a8c0 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.717+0000 7fb2cec00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c8071fd0 con 0x7fb2b0004800 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 0x7fb2c81f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 0x7fb2c81f56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55258/0 (socket says 172.21.15.163:55258) 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 -- 172.21.15.163:0/2816525679 learned_addr learned my addr 172.21.15.163:0/2816525679 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2cd800640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2b0004800 0x7fb2c814bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2cce00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c814a8c0 0x7fb2c8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 -- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2b0004800 msgr2=0x7fb2c814bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2b0004800 0x7fb2c814bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 -- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c814a8c0 msgr2=0x7fb2c8155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c814a8c0 0x7fb2c8155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 -- 172.21.15.163:0/2816525679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c814ae60 con 0x7fb2c8156150 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2cd800640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2b0004800 0x7fb2c814bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:25.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.718+0000 7fb2ce200640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 0x7fb2c81f56f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80b1dd0 tx=0x7fb2c000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.719+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0016020 con 0x7fb2c8156150 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.719+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2c0012920 con 0x7fb2c8156150 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.719+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c814b140 con 0x7fb2c8156150 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.719+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0012d70 con 0x7fb2c8156150 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.719+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c814b700 con 0x7fb2c8156150 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.720+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c80ce120 con 0x7fb2c8156150 2024-07-26T06:18:25.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.721+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb2c0019020 con 0x7fb2c8156150 2024-07-26T06:18:25.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.722+0000 7fb2c5600640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2a8076420 0x7fb2a80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:25.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.722+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb2c00974e0 con 0x7fb2c8156150 2024-07-26T06:18:25.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.722+0000 7fb2cd800640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2a8076420 0x7fb2a80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:25.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.723+0000 7fb2cd800640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2a8076420 0x7fb2a80788e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc007a60 tx=0x7fb2bc009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:25.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.726+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb2c0020030 con 0x7fb2c8156150 2024-07-26T06:18:25.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.932+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2c80b4670 con 0x7fb2c8156150 2024-07-26T06:18:25.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.934+0000 7fb2c5600640 1 -- 172.21.15.163:0/2816525679 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16684 (secure 0 0 0) 0x7fb2c0060d20 con 0x7fb2c8156150 2024-07-26T06:18:25.934 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:25.935 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":81,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:22.986998+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:19.305919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:20.414373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2a8076420 msgr2=0x7fb2a80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2a8076420 0x7fb2a80788e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc007a60 tx=0x7fb2bc009040 comp rx=0 tx=0).stop 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 msgr2=0x7fb2c81f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 0x7fb2c81f56f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80b1dd0 tx=0x7fb2c000c960 comp rx=0 tx=0).stop 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 shutdown_connections 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb2a8076420 0x7fb2a80788e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8156150 0x7fb2c81f56f0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c814a8c0 0x7fb2c8155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 --2- 172.21.15.163:0/2816525679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2b0004800 0x7fb2c814bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:25.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 >> 172.21.15.163:0/2816525679 conn(0x7fb2c80b0e30 msgr2=0x7fb2c815d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:25.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 shutdown_connections 2024-07-26T06:18:25.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:25.937+0000 7fb2cec00640 1 -- 172.21.15.163:0/2816525679 wait complete. 2024-07-26T06:18:25.956 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:18:25.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:18:26.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02b200640 1 -- 172.21.15.163:0/1913521606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc00c004d40 msgr2=0x7fc00c0051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02b200640 1 --2- 172.21.15.163:0/1913521606 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc00c004d40 0x7fc00c0051a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc024005fd0 tx=0x7fc00c005830 comp rx=0 tx=0).stop 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02a800640 1 -- 172.21.15.163:0/1913521606 reap_dead start 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02b200640 1 -- 172.21.15.163:0/1913521606 shutdown_connections 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02b200640 1 -- 172.21.15.163:0/1913521606 >> 172.21.15.163:0/1913521606 conn(0x7fc02c0b0e30 msgr2=0x7fc02c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02b200640 1 -- 172.21.15.163:0/1913521606 shutdown_connections 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.160+0000 7fc02b200640 1 -- 172.21.15.163:0/1913521606 wait complete. 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.162+0000 7fc02b200640 1 Processor -- start 2024-07-26T06:18:26.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.162+0000 7fc02b200640 1 -- start start 2024-07-26T06:18:26.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc02c149ba0 0x7fc02c14ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc02c14d140 0x7fc02c150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 0x7fc02c0b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c159370 con 0x7fc02c0b2130 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c06f500 con 0x7fc02c14d140 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc02c1594f0 con 0x7fc02c149ba0 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 0x7fc02c0b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 0x7fc02c0b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55264/0 (socket says 172.21.15.163:55264) 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc02a800640 1 -- 172.21.15.163:0/922226027 learned_addr learned my addr 172.21.15.163:0/922226027 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.163+0000 7fc029e00640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc02c149ba0 0x7fc02c14ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc023e00640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc02c14d140 0x7fc02c150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02a800640 1 -- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc02c149ba0 msgr2=0x7fc02c14ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02a800640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc02c149ba0 0x7fc02c14ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02a800640 1 -- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc02c14d140 msgr2=0x7fc02c150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02a800640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc02c14d140 0x7fc02c150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02a800640 1 -- 172.21.15.163:0/922226027 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c149f80 con 0x7fc02c0b2130 2024-07-26T06:18:26.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02a800640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 0x7fc02c0b25b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc02402b860 tx=0x7fc024005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:26.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024043070 con 0x7fc02c0b2130 2024-07-26T06:18:26.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc024034c40 con 0x7fc02c0b2130 2024-07-26T06:18:26.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024038040 con 0x7fc02c0b2130 2024-07-26T06:18:26.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02c0742d0 con 0x7fc02c0b2130 2024-07-26T06:18:26.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.164+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc02c074570 con 0x7fc02c0b2130 2024-07-26T06:18:26.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.166+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc02c0cd530 con 0x7fc02c0b2130 2024-07-26T06:18:26.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.167+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fc024046070 con 0x7fc02c0b2130 2024-07-26T06:18:26.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.168+0000 7fc022000640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc008076420 0x7fc0080788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.168+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc0240b7740 con 0x7fc02c0b2130 2024-07-26T06:18:26.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.168+0000 7fc029e00640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc008076420 0x7fc0080788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:26.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.169+0000 7fc029e00640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc008076420 0x7fc0080788e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc01c001eb0 tx=0x7fc01c009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:26.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.172+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0240808b0 con 0x7fc02c0b2130 2024-07-26T06:18:26.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.360+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc02c14dad0 con 0x7fc008076420 2024-07-26T06:18:26.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.367+0000 7fc022000640 1 -- 172.21.15.163:0/922226027 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240176 (secure 0 0 0) 0x7fc01c0050d0 con 0x7fc008076420 2024-07-26T06:18:26.369 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:26.372 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc008076420 msgr2=0x7fc0080788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc008076420 0x7fc0080788e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc01c001eb0 tx=0x7fc01c009040 comp rx=0 tx=0).stop 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 msgr2=0x7fc02c0b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 0x7fc02c0b25b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc02402b860 tx=0x7fc024005fd0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 shutdown_connections 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc008076420 0x7fc0080788e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc02c0b2130 0x7fc02c0b25b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc02c14d140 0x7fc02c150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 --2- 172.21.15.163:0/922226027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc02c149ba0 0x7fc02c14ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 >> 172.21.15.163:0/922226027 conn(0x7fc02c0b0e30 msgr2=0x7fc02c154760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:26.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 shutdown_connections 2024-07-26T06:18:26.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.373+0000 7fc02b200640 1 -- 172.21.15.163:0/922226027 wait complete. 2024-07-26T06:18:26.391 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-07-26T06:18:24.462839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257984,"kb_used_data":8512,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716736,"statfs":{"total":773094113280,"available":772829937664,"internally_reserved":0,"allocated":8716288,"data_stored":4368418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.518273"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.958347+0000","last_change":"2024-07-26T06:18:19.958347+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.958347+0000","last_undegraded":"2024-07-26T06:18:19.958347+0000","last_fullsized":"2024-07-26T06:18:19.958347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.950843+0000","last_change":"2024-07-26T06:18:19.950843+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.950843+0000","last_undegraded":"2024-07-26T06:18:19.950843+0000","last_fullsized":"2024-07-26T06:18:19.950843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.955428+0000","last_change":"2024-07-26T06:18:19.955428+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.955428+0000","last_undegraded":"2024-07-26T06:18:19.955428+0000","last_fullsized":"2024-07-26T06:18:19.955428+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.955281+0000","last_change":"2024-07-26T06:18:19.955281+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.955281+0000","last_undegraded":"2024-07-26T06:18:19.955281+0000","last_fullsized":"2024-07-26T06:18:19.955281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.950865+0000","last_change":"2024-07-26T06:18:19.950865+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.950865+0000","last_undegraded":"2024-07-26T06:18:19.950865+0000","last_fullsized":"2024-07-26T06:18:19.950865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.953578+0000","last_change":"2024-07-26T06:18:19.953578+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.953578+0000","last_undegraded":"2024-07-26T06:18:19.953578+0000","last_fullsized":"2024-07-26T06:18:19.953578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.957404+0000","last_change":"2024-07-26T06:18:19.957404+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.957404+0000","last_undegraded":"2024-07-26T06:18:19.957404+0000","last_fullsized":"2024-07-26T06:18:19.957404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.957962+0000","last_change":"2024-07-26T06:18:19.957962+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.957962+0000","last_undegraded":"2024-07-26T06:18:19.957962+0000","last_fullsized":"2024-07-26T06:18:19.957962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.956949+0000","last_change":"2024-07-26T06:18:19.956949+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.956949+0000","last_undegraded":"2024-07-26T06:18:19.956949+0000","last_fullsized":"2024-07-26T06:18:19.956949+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.952100+0000","last_change":"2024-07-26T06:18:19.952100+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.952100+0000","last_undegraded":"2024-07-26T06:18:19.952100+0000","last_fullsized":"2024-07-26T06:18:19.952100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.952514+0000","last_change":"2024-07-26T06:18:19.952514+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.952514+0000","last_undegraded":"2024-07-26T06:18:19.952514+0000","last_fullsized":"2024-07-26T06:18:19.952514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.963543+0000","last_change":"2024-07-26T06:18:19.963543+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.963543+0000","last_undegraded":"2024-07-26T06:18:19.963543+0000","last_fullsized":"2024-07-26T06:18:19.963543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.956974+0000","last_change":"2024-07-26T06:18:19.956974+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.956974+0000","last_undegraded":"2024-07-26T06:18:19.956974+0000","last_fullsized":"2024-07-26T06:18:19.956974+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.957564+0000","last_change":"2024-07-26T06:18:19.957564+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.957564+0000","last_undegraded":"2024-07-26T06:18:19.957564+0000","last_fullsized":"2024-07-26T06:18:19.957564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.965314+0000","last_change":"2024-07-26T06:18:19.965314+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.965314+0000","last_undegraded":"2024-07-26T06:18:19.965314+0000","last_fullsized":"2024-07-26T06:18:19.965314+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.954689+0000","last_change":"2024-07-26T06:18:19.954689+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.954689+0000","last_undegraded":"2024-07-26T06:18:19.954689+0000","last_fullsized":"2024-07-26T06:18:19.954689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.952764+0000","last_change":"2024-07-26T06:18:19.952764+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.952764+0000","last_undegraded":"2024-07-26T06:18:19.952764+0000","last_fullsized":"2024-07-26T06:18:19.952764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.956273+0000","last_change":"2024-07-26T06:18:19.956273+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.956273+0000","last_undegraded":"2024-07-26T06:18:19.956273+0000","last_fullsized":"2024-07-26T06:18:19.956273+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.954477+0000","last_change":"2024-07-26T06:18:19.954477+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.954477+0000","last_undegraded":"2024-07-26T06:18:19.954477+0000","last_fullsized":"2024-07-26T06:18:19.954477+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.956582+0000","last_change":"2024-07-26T06:18:19.956582+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.956582+0000","last_undegraded":"2024-07-26T06:18:19.956582+0000","last_fullsized":"2024-07-26T06:18:19.956582+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.950673+0000","last_change":"2024-07-26T06:18:19.950673+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.950673+0000","last_undegraded":"2024-07-26T06:18:19.950673+0000","last_fullsized":"2024-07-26T06:18:19.950673+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.950750+0000","last_change":"2024-07-26T06:18:19.950750+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.950750+0000","last_undegraded":"2024-07-26T06:18:19.950750+0000","last_fullsized":"2024-07-26T06:18:19.950750+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.956503+0000","last_change":"2024-07-26T06:18:19.956503+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.956503+0000","last_undegraded":"2024-07-26T06:18:19.956503+0000","last_fullsized":"2024-07-26T06:18:19.956503+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.960020+0000","last_change":"2024-07-26T06:18:19.960020+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.960020+0000","last_undegraded":"2024-07-26T06:18:19.960020+0000","last_fullsized":"2024-07-26T06:18:19.960020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.960171+0000","last_change":"2024-07-26T06:18:19.960171+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.960171+0000","last_undegraded":"2024-07-26T06:18:19.960171+0000","last_fullsized":"2024-07-26T06:18:19.960171+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.955594+0000","last_change":"2024-07-26T06:18:19.955594+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.955594+0000","last_undegraded":"2024-07-26T06:18:19.955594+0000","last_fullsized":"2024-07-26T06:18:19.955594+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.952720+0000","last_change":"2024-07-26T06:18:19.952720+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.952720+0000","last_undegraded":"2024-07-26T06:18:19.952720+0000","last_fullsized":"2024-07-26T06:18:19.952720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.956132+0000","last_change":"2024-07-26T06:18:19.956132+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.956132+0000","last_undegraded":"2024-07-26T06:18:19.956132+0000","last_fullsized":"2024-07-26T06:18:19.956132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.950751+0000","last_change":"2024-07-26T06:18:19.950751+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.950751+0000","last_undegraded":"2024-07-26T06:18:19.950751+0000","last_fullsized":"2024-07-26T06:18:19.950751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.952744+0000","last_change":"2024-07-26T06:18:19.952744+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.952744+0000","last_undegraded":"2024-07-26T06:18:19.952744+0000","last_fullsized":"2024-07-26T06:18:19.952744+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.950818+0000","last_change":"2024-07-26T06:18:19.950818+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.950818+0000","last_undegraded":"2024-07-26T06:18:19.950818+0000","last_fullsized":"2024-07-26T06:18:19.950818+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-07-26T06:18:20.944385+0000","last_change":"2024-07-26T06:18:20.944385+0000","last_active":"2024-07-26T06:18:20.944385+0000","last_peered":"2024-07-26T06:18:20.944385+0000","last_clean":"2024-07-26T06:18:20.944385+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:20.944385+0000","last_undegraded":"2024-07-26T06:18:20.944385+0000","last_fullsized":"2024-07-26T06:18:20.944385+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-07-26T06:18:19.961934+0000","last_change":"2024-07-26T06:18:19.961934+0000","last_active":"2024-07-26T06:18:19.937895+0000","last_peered":"2024-07-26T06:18:19.937895+0000","last_clean":"2024-07-26T06:18:19.937895+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-07-26T06:18:19.961934+0000","last_undegraded":"2024-07-26T06:18:19.961934+0000","last_fullsized":"2024-07-26T06:18:19.961934+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"70'79","reported_seq":239,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-07-26T06:18:19.963368+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:18:19.963368+0000","last_peered":"2024-07-26T06:18:19.963368+0000","last_clean":"2024-07-26T06:18:19.963368+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:18:19.963368+0000","last_undegraded":"2024-07-26T06:18:19.963368+0000","last_fullsized":"2024-07-26T06:18:19.963368+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509503,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1544192,"data_stored":993191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":397006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509503,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":950272,"data_stored":402823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542207,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":937984,"data_stored":397006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542206,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":397006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509505,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":937984,"data_stored":397006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":937984,"data_stored":397006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509504,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1531904,"data_stored":987374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:18:26.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:18:26.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56b200640 1 -- 172.21.15.163:0/3621309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 msgr2=0x7ff54c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56b200640 1 --2- 172.21.15.163:0/3621309509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff54c00b9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c5c0 tx=0x7ff54c005830 comp rx=0 tx=0).stop 2024-07-26T06:18:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56a800640 1 -- 172.21.15.163:0/3621309509 reap_dead start 2024-07-26T06:18:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56b200640 1 -- 172.21.15.163:0/3621309509 shutdown_connections 2024-07-26T06:18:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56b200640 1 -- 172.21.15.163:0/3621309509 >> 172.21.15.163:0/3621309509 conn(0x7ff5640b0e30 msgr2=0x7ff5640b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56b200640 1 -- 172.21.15.163:0/3621309509 shutdown_connections 2024-07-26T06:18:26.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.631+0000 7ff56b200640 1 -- 172.21.15.163:0/3621309509 wait complete. 2024-07-26T06:18:26.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.633+0000 7ff56b200640 1 Processor -- start 2024-07-26T06:18:26.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.633+0000 7ff56b200640 1 -- start start 2024-07-26T06:18:26.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff56b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff5641490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff56b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff564151fa0 0x7ff564153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff56b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5641535a0 0x7ff5640af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff56b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff56415a1d0 con 0x7ff54c004e60 2024-07-26T06:18:26.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff56b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff56415a350 con 0x7ff5641535a0 2024-07-26T06:18:26.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff56b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff5640743e0 con 0x7ff564151fa0 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff569e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff5641490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff569e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff5641490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55282/0 (socket says 172.21.15.163:55282) 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.634+0000 7ff569e00640 1 -- 172.21.15.163:0/2204704057 learned_addr learned my addr 172.21.15.163:0/2204704057 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff569e00640 1 -- 172.21.15.163:0/2204704057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff564151fa0 msgr2=0x7ff564153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff569e00640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff564151fa0 0x7ff564153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff569e00640 1 -- 172.21.15.163:0/2204704057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5641535a0 msgr2=0x7ff5640af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff569e00640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5641535a0 0x7ff5640af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff569e00640 1 -- 172.21.15.163:0/2204704057 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff564150ac0 con 0x7ff54c004e60 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff569e00640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff5641490e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff55800fa00 tx=0x7ff55800fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55800bc30 con 0x7ff54c004e60 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff558011440 con 0x7ff54c004e60 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5641511e0 con 0x7ff54c004e60 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558011870 con 0x7ff54c004e60 2024-07-26T06:18:26.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.635+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5641517a0 con 0x7ff54c004e60 2024-07-26T06:18:26.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.637+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5641539c0 con 0x7ff54c004e60 2024-07-26T06:18:26.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.637+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7ff558019020 con 0x7ff54c004e60 2024-07-26T06:18:26.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.638+0000 7ff55e000640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff548076420 0x7ff5480788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:26.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.638+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7ff558098350 con 0x7ff54c004e60 2024-07-26T06:18:26.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.639+0000 7ff569400640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff548076420 0x7ff5480788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:26.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.639+0000 7ff569400640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff548076420 0x7ff5480788e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff564154f00 tx=0x7ff554006940 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:26.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.642+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff5641539c0 con 0x7ff54c004e60 2024-07-26T06:18:26.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.845+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5640b26d0 con 0x7ff548076420 2024-07-26T06:18:26.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.857+0000 7ff55e000640 1 -- 172.21.15.163:0/2204704057 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7ff5640b26d0 con 0x7ff548076420 2024-07-26T06:18:26.860 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:26.862 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:18:26.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff548076420 msgr2=0x7ff5480788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff548076420 0x7ff5480788e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff564154f00 tx=0x7ff554006940 comp rx=0 tx=0).stop 2024-07-26T06:18:26.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 msgr2=0x7ff5641490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff5641490e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff55800fa00 tx=0x7ff55800fec0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 shutdown_connections 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff548076420 0x7ff5480788e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.862+0000 7ff56b200640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5641535a0 0x7ff5640af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.863+0000 7ff56b200640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff564151fa0 0x7ff564153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.863+0000 7ff56b200640 1 --2- 172.21.15.163:0/2204704057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff54c004e60 0x7ff5641490e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.863+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 >> 172.21.15.163:0/2204704057 conn(0x7ff5640b0e30 msgr2=0x7ff5640cd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.863+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 shutdown_connections 2024-07-26T06:18:26.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:26.863+0000 7ff56b200640 1 -- 172.21.15.163:0/2204704057 wait complete. 2024-07-26T06:18:26.881 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-07-26T06:18:26.463522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258428,"kb_used_data":8936,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716292,"statfs":{"total":773094113280,"available":772829483008,"internally_reserved":0,"allocated":9150464,"data_stored":4562940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001784"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049747+0000","last_change":"2024-07-26T06:18:21.987384+0000","last_active":"2024-07-26T06:18:23.049747+0000","last_peered":"2024-07-26T06:18:23.049747+0000","last_clean":"2024-07-26T06:18:23.049747+0000","last_became_active":"2024-07-26T06:18:21.986909+0000","last_became_peered":"2024-07-26T06:18:21.986909+0000","last_unstale":"2024-07-26T06:18:23.049747+0000","last_undegraded":"2024-07-26T06:18:23.049747+0000","last_fullsized":"2024-07-26T06:18:23.049747+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:59.016108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001387+0000","last_change":"2024-07-26T06:18:21.998182+0000","last_active":"2024-07-26T06:18:23.001387+0000","last_peered":"2024-07-26T06:18:23.001387+0000","last_clean":"2024-07-26T06:18:23.001387+0000","last_became_active":"2024-07-26T06:18:21.997814+0000","last_became_peered":"2024-07-26T06:18:21.997814+0000","last_unstale":"2024-07-26T06:18:23.001387+0000","last_undegraded":"2024-07-26T06:18:23.001387+0000","last_fullsized":"2024-07-26T06:18:23.001387+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:38.288258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049571+0000","last_change":"2024-07-26T06:18:22.008161+0000","last_active":"2024-07-26T06:18:23.049571+0000","last_peered":"2024-07-26T06:18:23.049571+0000","last_clean":"2024-07-26T06:18:23.049571+0000","last_became_active":"2024-07-26T06:18:22.007812+0000","last_became_peered":"2024-07-26T06:18:22.007812+0000","last_unstale":"2024-07-26T06:18:23.049571+0000","last_undegraded":"2024-07-26T06:18:23.049571+0000","last_fullsized":"2024-07-26T06:18:23.049571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:09:29.167864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045670+0000","last_change":"2024-07-26T06:18:22.015529+0000","last_active":"2024-07-26T06:18:23.045670+0000","last_peered":"2024-07-26T06:18:23.045670+0000","last_clean":"2024-07-26T06:18:23.045670+0000","last_became_active":"2024-07-26T06:18:22.015190+0000","last_became_peered":"2024-07-26T06:18:22.015190+0000","last_unstale":"2024-07-26T06:18:23.045670+0000","last_undegraded":"2024-07-26T06:18:23.045670+0000","last_fullsized":"2024-07-26T06:18:23.045670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:28.156647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002397+0000","last_change":"2024-07-26T06:18:21.998546+0000","last_active":"2024-07-26T06:18:23.002397+0000","last_peered":"2024-07-26T06:18:23.002397+0000","last_clean":"2024-07-26T06:18:23.002397+0000","last_became_active":"2024-07-26T06:18:21.998263+0000","last_became_peered":"2024-07-26T06:18:21.998263+0000","last_unstale":"2024-07-26T06:18:23.002397+0000","last_undegraded":"2024-07-26T06:18:23.002397+0000","last_fullsized":"2024-07-26T06:18:23.002397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:50.958358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046647+0000","last_change":"2024-07-26T06:18:22.009330+0000","last_active":"2024-07-26T06:18:23.046647+0000","last_peered":"2024-07-26T06:18:23.046647+0000","last_clean":"2024-07-26T06:18:23.046647+0000","last_became_active":"2024-07-26T06:18:22.008564+0000","last_became_peered":"2024-07-26T06:18:22.008564+0000","last_unstale":"2024-07-26T06:18:23.046647+0000","last_undegraded":"2024-07-26T06:18:23.046647+0000","last_fullsized":"2024-07-26T06:18:23.046647+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:38:16.849765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046316+0000","last_change":"2024-07-26T06:18:22.004817+0000","last_active":"2024-07-26T06:18:23.046316+0000","last_peered":"2024-07-26T06:18:23.046316+0000","last_clean":"2024-07-26T06:18:23.046316+0000","last_became_active":"2024-07-26T06:18:22.004511+0000","last_became_peered":"2024-07-26T06:18:22.004511+0000","last_unstale":"2024-07-26T06:18:23.046316+0000","last_undegraded":"2024-07-26T06:18:23.046316+0000","last_fullsized":"2024-07-26T06:18:23.046316+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:02.856155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046894+0000","last_change":"2024-07-26T06:18:22.021843+0000","last_active":"2024-07-26T06:18:23.046894+0000","last_peered":"2024-07-26T06:18:23.046894+0000","last_clean":"2024-07-26T06:18:23.046894+0000","last_became_active":"2024-07-26T06:18:22.021501+0000","last_became_peered":"2024-07-26T06:18:22.021501+0000","last_unstale":"2024-07-26T06:18:23.046894+0000","last_undegraded":"2024-07-26T06:18:23.046894+0000","last_fullsized":"2024-07-26T06:18:23.046894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:46:34.088512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049146+0000","last_change":"2024-07-26T06:18:22.014912+0000","last_active":"2024-07-26T06:18:23.049146+0000","last_peered":"2024-07-26T06:18:23.049146+0000","last_clean":"2024-07-26T06:18:23.049146+0000","last_became_active":"2024-07-26T06:18:22.014696+0000","last_became_peered":"2024-07-26T06:18:22.014696+0000","last_unstale":"2024-07-26T06:18:23.049146+0000","last_undegraded":"2024-07-26T06:18:23.049146+0000","last_fullsized":"2024-07-26T06:18:23.049146+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:23:12.657539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048755+0000","last_change":"2024-07-26T06:18:22.008828+0000","last_active":"2024-07-26T06:18:23.048755+0000","last_peered":"2024-07-26T06:18:23.048755+0000","last_clean":"2024-07-26T06:18:23.048755+0000","last_became_active":"2024-07-26T06:18:22.008404+0000","last_became_peered":"2024-07-26T06:18:22.008404+0000","last_unstale":"2024-07-26T06:18:23.048755+0000","last_undegraded":"2024-07-26T06:18:23.048755+0000","last_fullsized":"2024-07-26T06:18:23.048755+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:09:29.167864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001509+0000","last_change":"2024-07-26T06:18:22.016418+0000","last_active":"2024-07-26T06:18:23.001509+0000","last_peered":"2024-07-26T06:18:23.001509+0000","last_clean":"2024-07-26T06:18:23.001509+0000","last_became_active":"2024-07-26T06:18:22.016167+0000","last_became_peered":"2024-07-26T06:18:22.016167+0000","last_unstale":"2024-07-26T06:18:23.001509+0000","last_undegraded":"2024-07-26T06:18:23.001509+0000","last_fullsized":"2024-07-26T06:18:23.001509+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:37:10.942201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047844+0000","last_change":"2024-07-26T06:18:22.002437+0000","last_active":"2024-07-26T06:18:23.047844+0000","last_peered":"2024-07-26T06:18:23.047844+0000","last_clean":"2024-07-26T06:18:23.047844+0000","last_became_active":"2024-07-26T06:18:22.002053+0000","last_became_peered":"2024-07-26T06:18:22.002053+0000","last_unstale":"2024-07-26T06:18:23.047844+0000","last_undegraded":"2024-07-26T06:18:23.047844+0000","last_fullsized":"2024-07-26T06:18:23.047844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:58:51.096745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049279+0000","last_change":"2024-07-26T06:18:22.001029+0000","last_active":"2024-07-26T06:18:23.049279+0000","last_peered":"2024-07-26T06:18:23.049279+0000","last_clean":"2024-07-26T06:18:23.049279+0000","last_became_active":"2024-07-26T06:18:22.000687+0000","last_became_peered":"2024-07-26T06:18:22.000687+0000","last_unstale":"2024-07-26T06:18:23.049279+0000","last_undegraded":"2024-07-26T06:18:23.049279+0000","last_fullsized":"2024-07-26T06:18:23.049279+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:51.949797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045838+0000","last_change":"2024-07-26T06:18:22.003985+0000","last_active":"2024-07-26T06:18:23.045838+0000","last_peered":"2024-07-26T06:18:23.045838+0000","last_clean":"2024-07-26T06:18:23.045838+0000","last_became_active":"2024-07-26T06:18:22.003638+0000","last_became_peered":"2024-07-26T06:18:22.003638+0000","last_unstale":"2024-07-26T06:18:23.045838+0000","last_undegraded":"2024-07-26T06:18:23.045838+0000","last_fullsized":"2024-07-26T06:18:23.045838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:22:09.271916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002779+0000","last_change":"2024-07-26T06:18:22.006967+0000","last_active":"2024-07-26T06:18:23.002779+0000","last_peered":"2024-07-26T06:18:23.002779+0000","last_clean":"2024-07-26T06:18:23.002779+0000","last_became_active":"2024-07-26T06:18:22.006724+0000","last_became_peered":"2024-07-26T06:18:22.006724+0000","last_unstale":"2024-07-26T06:18:23.002779+0000","last_undegraded":"2024-07-26T06:18:23.002779+0000","last_fullsized":"2024-07-26T06:18:23.002779+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:26:45.584256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002110+0000","last_change":"2024-07-26T06:18:21.989516+0000","last_active":"2024-07-26T06:18:23.002110+0000","last_peered":"2024-07-26T06:18:23.002110+0000","last_clean":"2024-07-26T06:18:23.002110+0000","last_became_active":"2024-07-26T06:18:21.989037+0000","last_became_peered":"2024-07-26T06:18:21.989037+0000","last_unstale":"2024-07-26T06:18:23.002110+0000","last_undegraded":"2024-07-26T06:18:23.002110+0000","last_fullsized":"2024-07-26T06:18:23.002110+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:25.722672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.003023+0000","last_change":"2024-07-26T06:18:21.995947+0000","last_active":"2024-07-26T06:18:23.003023+0000","last_peered":"2024-07-26T06:18:23.003023+0000","last_clean":"2024-07-26T06:18:23.003023+0000","last_became_active":"2024-07-26T06:18:21.995565+0000","last_became_peered":"2024-07-26T06:18:21.995565+0000","last_unstale":"2024-07-26T06:18:23.003023+0000","last_undegraded":"2024-07-26T06:18:23.003023+0000","last_fullsized":"2024-07-26T06:18:23.003023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:55:37.212648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046341+0000","last_change":"2024-07-26T06:18:22.004882+0000","last_active":"2024-07-26T06:18:23.046341+0000","last_peered":"2024-07-26T06:18:23.046341+0000","last_clean":"2024-07-26T06:18:23.046341+0000","last_became_active":"2024-07-26T06:18:22.004573+0000","last_became_peered":"2024-07-26T06:18:22.004573+0000","last_unstale":"2024-07-26T06:18:23.046341+0000","last_undegraded":"2024-07-26T06:18:23.046341+0000","last_fullsized":"2024-07-26T06:18:23.046341+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:10.950765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010995+0000","last_change":"2024-07-26T06:18:22.005096+0000","last_active":"2024-07-26T06:18:23.010995+0000","last_peered":"2024-07-26T06:18:23.010995+0000","last_clean":"2024-07-26T06:18:23.010995+0000","last_became_active":"2024-07-26T06:18:22.004740+0000","last_became_peered":"2024-07-26T06:18:22.004740+0000","last_unstale":"2024-07-26T06:18:23.010995+0000","last_undegraded":"2024-07-26T06:18:23.010995+0000","last_fullsized":"2024-07-26T06:18:23.010995+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:51.949797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045840+0000","last_change":"2024-07-26T06:18:22.001348+0000","last_active":"2024-07-26T06:18:23.045840+0000","last_peered":"2024-07-26T06:18:23.045840+0000","last_clean":"2024-07-26T06:18:23.045840+0000","last_became_active":"2024-07-26T06:18:21.999177+0000","last_became_peered":"2024-07-26T06:18:21.999177+0000","last_unstale":"2024-07-26T06:18:23.045840+0000","last_undegraded":"2024-07-26T06:18:23.045840+0000","last_fullsized":"2024-07-26T06:18:23.045840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:55.110986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045845+0000","last_change":"2024-07-26T06:18:22.002890+0000","last_active":"2024-07-26T06:18:23.045845+0000","last_peered":"2024-07-26T06:18:23.045845+0000","last_clean":"2024-07-26T06:18:23.045845+0000","last_became_active":"2024-07-26T06:18:22.001980+0000","last_became_peered":"2024-07-26T06:18:22.001980+0000","last_unstale":"2024-07-26T06:18:23.045845+0000","last_undegraded":"2024-07-26T06:18:23.045845+0000","last_fullsized":"2024-07-26T06:18:23.045845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:05:49.763270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000750+0000","last_change":"2024-07-26T06:18:22.006686+0000","last_active":"2024-07-26T06:18:23.000750+0000","last_peered":"2024-07-26T06:18:23.000750+0000","last_clean":"2024-07-26T06:18:23.000750+0000","last_became_active":"2024-07-26T06:18:22.006219+0000","last_became_peered":"2024-07-26T06:18:22.006219+0000","last_unstale":"2024-07-26T06:18:23.000750+0000","last_undegraded":"2024-07-26T06:18:23.000750+0000","last_fullsized":"2024-07-26T06:18:23.000750+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:28.095148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047579+0000","last_change":"2024-07-26T06:18:21.998811+0000","last_active":"2024-07-26T06:18:23.047579+0000","last_peered":"2024-07-26T06:18:23.047579+0000","last_clean":"2024-07-26T06:18:23.047579+0000","last_became_active":"2024-07-26T06:18:21.997458+0000","last_became_peered":"2024-07-26T06:18:21.997458+0000","last_unstale":"2024-07-26T06:18:23.047579+0000","last_undegraded":"2024-07-26T06:18:23.047579+0000","last_fullsized":"2024-07-26T06:18:23.047579+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:52:05.787110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045672+0000","last_change":"2024-07-26T06:18:22.003311+0000","last_active":"2024-07-26T06:18:23.045672+0000","last_peered":"2024-07-26T06:18:23.045672+0000","last_clean":"2024-07-26T06:18:23.045672+0000","last_became_active":"2024-07-26T06:18:22.002332+0000","last_became_peered":"2024-07-26T06:18:22.002332+0000","last_unstale":"2024-07-26T06:18:23.045672+0000","last_undegraded":"2024-07-26T06:18:23.045672+0000","last_fullsized":"2024-07-26T06:18:23.045672+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:56:11.297984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001064+0000","last_change":"2024-07-26T06:18:21.999277+0000","last_active":"2024-07-26T06:18:23.001064+0000","last_peered":"2024-07-26T06:18:23.001064+0000","last_clean":"2024-07-26T06:18:23.001064+0000","last_became_active":"2024-07-26T06:18:21.998195+0000","last_became_peered":"2024-07-26T06:18:21.998195+0000","last_unstale":"2024-07-26T06:18:23.001064+0000","last_undegraded":"2024-07-26T06:18:23.001064+0000","last_fullsized":"2024-07-26T06:18:23.001064+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:02:27.532204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049735+0000","last_change":"2024-07-26T06:18:22.005073+0000","last_active":"2024-07-26T06:18:23.049735+0000","last_peered":"2024-07-26T06:18:23.049735+0000","last_clean":"2024-07-26T06:18:23.049735+0000","last_became_active":"2024-07-26T06:18:22.004785+0000","last_became_peered":"2024-07-26T06:18:22.004785+0000","last_unstale":"2024-07-26T06:18:23.049735+0000","last_undegraded":"2024-07-26T06:18:23.049735+0000","last_fullsized":"2024-07-26T06:18:23.049735+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:20:13.729479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002700+0000","last_change":"2024-07-26T06:18:22.001069+0000","last_active":"2024-07-26T06:18:23.002700+0000","last_peered":"2024-07-26T06:18:23.002700+0000","last_clean":"2024-07-26T06:18:23.002700+0000","last_became_active":"2024-07-26T06:18:22.000768+0000","last_became_peered":"2024-07-26T06:18:22.000768+0000","last_unstale":"2024-07-26T06:18:23.002700+0000","last_undegraded":"2024-07-26T06:18:23.002700+0000","last_fullsized":"2024-07-26T06:18:23.002700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:50:09.894858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050889+0000","last_change":"2024-07-26T06:18:21.989127+0000","last_active":"2024-07-26T06:18:23.050889+0000","last_peered":"2024-07-26T06:18:23.050889+0000","last_clean":"2024-07-26T06:18:23.050889+0000","last_became_active":"2024-07-26T06:18:21.988469+0000","last_became_peered":"2024-07-26T06:18:21.988469+0000","last_unstale":"2024-07-26T06:18:23.050889+0000","last_undegraded":"2024-07-26T06:18:23.050889+0000","last_fullsized":"2024-07-26T06:18:23.050889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:32:12.907681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001710+0000","last_change":"2024-07-26T06:18:22.015688+0000","last_active":"2024-07-26T06:18:23.001710+0000","last_peered":"2024-07-26T06:18:23.001710+0000","last_clean":"2024-07-26T06:18:23.001710+0000","last_became_active":"2024-07-26T06:18:22.015405+0000","last_became_peered":"2024-07-26T06:18:22.015405+0000","last_unstale":"2024-07-26T06:18:23.001710+0000","last_undegraded":"2024-07-26T06:18:23.001710+0000","last_fullsized":"2024-07-26T06:18:23.001710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:24:23.127466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000820+0000","last_change":"2024-07-26T06:18:22.006231+0000","last_active":"2024-07-26T06:18:23.000820+0000","last_peered":"2024-07-26T06:18:23.000820+0000","last_clean":"2024-07-26T06:18:23.000820+0000","last_became_active":"2024-07-26T06:18:22.005444+0000","last_became_peered":"2024-07-26T06:18:22.005444+0000","last_unstale":"2024-07-26T06:18:23.000820+0000","last_undegraded":"2024-07-26T06:18:23.000820+0000","last_fullsized":"2024-07-26T06:18:23.000820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:12:44.404784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002053+0000","last_change":"2024-07-26T06:18:22.003194+0000","last_active":"2024-07-26T06:18:23.002053+0000","last_peered":"2024-07-26T06:18:23.002053+0000","last_clean":"2024-07-26T06:18:23.002053+0000","last_became_active":"2024-07-26T06:18:22.002844+0000","last_became_peered":"2024-07-26T06:18:22.002844+0000","last_unstale":"2024-07-26T06:18:23.002053+0000","last_undegraded":"2024-07-26T06:18:23.002053+0000","last_fullsized":"2024-07-26T06:18:23.002053+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:40:26.105179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047265+0000","last_change":"2024-07-26T06:18:21.998704+0000","last_active":"2024-07-26T06:18:23.047265+0000","last_peered":"2024-07-26T06:18:23.047265+0000","last_clean":"2024-07-26T06:18:23.047265+0000","last_became_active":"2024-07-26T06:18:21.998384+0000","last_became_peered":"2024-07-26T06:18:21.998384+0000","last_unstale":"2024-07-26T06:18:23.047265+0000","last_undegraded":"2024-07-26T06:18:23.047265+0000","last_fullsized":"2024-07-26T06:18:23.047265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:34:45.374115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002938+0000","last_change":"2024-07-26T06:18:21.991300+0000","last_active":"2024-07-26T06:18:23.002938+0000","last_peered":"2024-07-26T06:18:23.002938+0000","last_clean":"2024-07-26T06:18:23.002938+0000","last_became_active":"2024-07-26T06:18:21.990671+0000","last_became_peered":"2024-07-26T06:18:21.990671+0000","last_unstale":"2024-07-26T06:18:23.002938+0000","last_undegraded":"2024-07-26T06:18:23.002938+0000","last_fullsized":"2024-07-26T06:18:23.002938+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:12.478299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050216+0000","last_change":"2024-07-26T06:18:21.000115+0000","last_active":"2024-07-26T06:18:23.050216+0000","last_peered":"2024-07-26T06:18:23.050216+0000","last_clean":"2024-07-26T06:18:23.050216+0000","last_became_active":"2024-07-26T06:18:20.999831+0000","last_became_peered":"2024-07-26T06:18:20.999831+0000","last_unstale":"2024-07-26T06:18:23.050216+0000","last_undegraded":"2024-07-26T06:18:23.050216+0000","last_fullsized":"2024-07-26T06:18:23.050216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:53.249280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050088+0000","last_change":"2024-07-26T06:18:21.986356+0000","last_active":"2024-07-26T06:18:23.050088+0000","last_peered":"2024-07-26T06:18:23.050088+0000","last_clean":"2024-07-26T06:18:23.050088+0000","last_became_active":"2024-07-26T06:18:21.985916+0000","last_became_peered":"2024-07-26T06:18:21.985916+0000","last_unstale":"2024-07-26T06:18:23.050088+0000","last_undegraded":"2024-07-26T06:18:23.050088+0000","last_fullsized":"2024-07-26T06:18:23.050088+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:20:13.729479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002866+0000","last_change":"2024-07-26T06:18:20.993818+0000","last_active":"2024-07-26T06:18:23.002866+0000","last_peered":"2024-07-26T06:18:23.002866+0000","last_clean":"2024-07-26T06:18:23.002866+0000","last_became_active":"2024-07-26T06:18:20.993544+0000","last_became_peered":"2024-07-26T06:18:20.993544+0000","last_unstale":"2024-07-26T06:18:23.002866+0000","last_undegraded":"2024-07-26T06:18:23.002866+0000","last_fullsized":"2024-07-26T06:18:23.002866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:24:22.120975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048241+0000","last_change":"2024-07-26T06:18:22.014112+0000","last_active":"2024-07-26T06:18:23.048241+0000","last_peered":"2024-07-26T06:18:23.048241+0000","last_clean":"2024-07-26T06:18:23.048241+0000","last_became_active":"2024-07-26T06:18:22.013788+0000","last_became_peered":"2024-07-26T06:18:22.013788+0000","last_unstale":"2024-07-26T06:18:23.048241+0000","last_undegraded":"2024-07-26T06:18:23.048241+0000","last_fullsized":"2024-07-26T06:18:23.048241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:41:36.941404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.011071+0000","last_change":"2024-07-26T06:18:20.991831+0000","last_active":"2024-07-26T06:18:23.011071+0000","last_peered":"2024-07-26T06:18:23.011071+0000","last_clean":"2024-07-26T06:18:23.011071+0000","last_became_active":"2024-07-26T06:18:20.991535+0000","last_became_peered":"2024-07-26T06:18:20.991535+0000","last_unstale":"2024-07-26T06:18:23.011071+0000","last_undegraded":"2024-07-26T06:18:23.011071+0000","last_fullsized":"2024-07-26T06:18:23.011071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:17.413979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050158+0000","last_change":"2024-07-26T06:18:22.004346+0000","last_active":"2024-07-26T06:18:23.050158+0000","last_peered":"2024-07-26T06:18:23.050158+0000","last_clean":"2024-07-26T06:18:23.050158+0000","last_became_active":"2024-07-26T06:18:22.004006+0000","last_became_peered":"2024-07-26T06:18:22.004006+0000","last_unstale":"2024-07-26T06:18:23.050158+0000","last_undegraded":"2024-07-26T06:18:23.050158+0000","last_fullsized":"2024-07-26T06:18:23.050158+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:05.857649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010809+0000","last_change":"2024-07-26T06:18:20.999165+0000","last_active":"2024-07-26T06:18:23.010809+0000","last_peered":"2024-07-26T06:18:23.010809+0000","last_clean":"2024-07-26T06:18:23.010809+0000","last_became_active":"2024-07-26T06:18:20.998732+0000","last_became_peered":"2024-07-26T06:18:20.998732+0000","last_unstale":"2024-07-26T06:18:23.010809+0000","last_undegraded":"2024-07-26T06:18:23.010809+0000","last_fullsized":"2024-07-26T06:18:23.010809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:40.257374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000953+0000","last_change":"2024-07-26T06:18:22.006470+0000","last_active":"2024-07-26T06:18:23.000953+0000","last_peered":"2024-07-26T06:18:23.000953+0000","last_clean":"2024-07-26T06:18:23.000953+0000","last_became_active":"2024-07-26T06:18:22.005011+0000","last_became_peered":"2024-07-26T06:18:22.005011+0000","last_unstale":"2024-07-26T06:18:23.000953+0000","last_undegraded":"2024-07-26T06:18:23.000953+0000","last_fullsized":"2024-07-26T06:18:23.000953+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:10.728261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002504+0000","last_change":"2024-07-26T06:18:20.999538+0000","last_active":"2024-07-26T06:18:23.002504+0000","last_peered":"2024-07-26T06:18:23.002504+0000","last_clean":"2024-07-26T06:18:23.002504+0000","last_became_active":"2024-07-26T06:18:20.998912+0000","last_became_peered":"2024-07-26T06:18:20.998912+0000","last_unstale":"2024-07-26T06:18:23.002504+0000","last_undegraded":"2024-07-26T06:18:23.002504+0000","last_fullsized":"2024-07-26T06:18:23.002504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:53:15.625863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002444+0000","last_change":"2024-07-26T06:18:22.001994+0000","last_active":"2024-07-26T06:18:23.002444+0000","last_peered":"2024-07-26T06:18:23.002444+0000","last_clean":"2024-07-26T06:18:23.002444+0000","last_became_active":"2024-07-26T06:18:22.001680+0000","last_became_peered":"2024-07-26T06:18:22.001680+0000","last_unstale":"2024-07-26T06:18:23.002444+0000","last_undegraded":"2024-07-26T06:18:23.002444+0000","last_fullsized":"2024-07-26T06:18:23.002444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:53:49.453541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002132+0000","last_change":"2024-07-26T06:18:20.994636+0000","last_active":"2024-07-26T06:18:23.002132+0000","last_peered":"2024-07-26T06:18:23.002132+0000","last_clean":"2024-07-26T06:18:23.002132+0000","last_became_active":"2024-07-26T06:18:20.994369+0000","last_became_peered":"2024-07-26T06:18:20.994369+0000","last_unstale":"2024-07-26T06:18:23.002132+0000","last_undegraded":"2024-07-26T06:18:23.002132+0000","last_fullsized":"2024-07-26T06:18:23.002132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:37.281768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049811+0000","last_change":"2024-07-26T06:18:22.005975+0000","last_active":"2024-07-26T06:18:23.049811+0000","last_peered":"2024-07-26T06:18:23.049811+0000","last_clean":"2024-07-26T06:18:23.049811+0000","last_became_active":"2024-07-26T06:18:22.005695+0000","last_became_peered":"2024-07-26T06:18:22.005695+0000","last_unstale":"2024-07-26T06:18:23.049811+0000","last_undegraded":"2024-07-26T06:18:23.049811+0000","last_fullsized":"2024-07-26T06:18:23.049811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:36:11.068267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010727+0000","last_change":"2024-07-26T06:18:21.000776+0000","last_active":"2024-07-26T06:18:23.010727+0000","last_peered":"2024-07-26T06:18:23.010727+0000","last_clean":"2024-07-26T06:18:23.010727+0000","last_became_active":"2024-07-26T06:18:20.999880+0000","last_became_peered":"2024-07-26T06:18:20.999880+0000","last_unstale":"2024-07-26T06:18:23.010727+0000","last_undegraded":"2024-07-26T06:18:23.010727+0000","last_fullsized":"2024-07-26T06:18:23.010727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:16.539912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002678+0000","last_change":"2024-07-26T06:18:22.009755+0000","last_active":"2024-07-26T06:18:23.002678+0000","last_peered":"2024-07-26T06:18:23.002678+0000","last_clean":"2024-07-26T06:18:23.002678+0000","last_became_active":"2024-07-26T06:18:22.009400+0000","last_became_peered":"2024-07-26T06:18:22.009400+0000","last_unstale":"2024-07-26T06:18:23.002678+0000","last_undegraded":"2024-07-26T06:18:23.002678+0000","last_fullsized":"2024-07-26T06:18:23.002678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:41:34.525926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000333+0000","last_change":"2024-07-26T06:18:20.996354+0000","last_active":"2024-07-26T06:18:23.000333+0000","last_peered":"2024-07-26T06:18:23.000333+0000","last_clean":"2024-07-26T06:18:23.000333+0000","last_became_active":"2024-07-26T06:18:20.995979+0000","last_became_peered":"2024-07-26T06:18:20.995979+0000","last_unstale":"2024-07-26T06:18:23.000333+0000","last_undegraded":"2024-07-26T06:18:23.000333+0000","last_fullsized":"2024-07-26T06:18:23.000333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:52.248251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.003262+0000","last_change":"2024-07-26T06:18:22.009729+0000","last_active":"2024-07-26T06:18:23.003262+0000","last_peered":"2024-07-26T06:18:23.003262+0000","last_clean":"2024-07-26T06:18:23.003262+0000","last_became_active":"2024-07-26T06:18:22.009475+0000","last_became_peered":"2024-07-26T06:18:22.009475+0000","last_unstale":"2024-07-26T06:18:23.003262+0000","last_undegraded":"2024-07-26T06:18:23.003262+0000","last_fullsized":"2024-07-26T06:18:23.003262+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:53:41.918378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050426+0000","last_change":"2024-07-26T06:18:20.995404+0000","last_active":"2024-07-26T06:18:23.050426+0000","last_peered":"2024-07-26T06:18:23.050426+0000","last_clean":"2024-07-26T06:18:23.050426+0000","last_became_active":"2024-07-26T06:18:20.994310+0000","last_became_peered":"2024-07-26T06:18:20.994310+0000","last_unstale":"2024-07-26T06:18:23.050426+0000","last_undegraded":"2024-07-26T06:18:23.050426+0000","last_fullsized":"2024-07-26T06:18:23.050426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:01:05.979196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050170+0000","last_change":"2024-07-26T06:18:22.009089+0000","last_active":"2024-07-26T06:18:23.050170+0000","last_peered":"2024-07-26T06:18:23.050170+0000","last_clean":"2024-07-26T06:18:23.050170+0000","last_became_active":"2024-07-26T06:18:22.008760+0000","last_became_peered":"2024-07-26T06:18:22.008760+0000","last_unstale":"2024-07-26T06:18:23.050170+0000","last_undegraded":"2024-07-26T06:18:23.050170+0000","last_fullsized":"2024-07-26T06:18:23.050170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:58:51.096745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048811+0000","last_change":"2024-07-26T06:18:20.995803+0000","last_active":"2024-07-26T06:18:23.048811+0000","last_peered":"2024-07-26T06:18:23.048811+0000","last_clean":"2024-07-26T06:18:23.048811+0000","last_became_active":"2024-07-26T06:18:20.994086+0000","last_became_peered":"2024-07-26T06:18:20.994086+0000","last_unstale":"2024-07-26T06:18:23.048811+0000","last_undegraded":"2024-07-26T06:18:23.048811+0000","last_fullsized":"2024-07-26T06:18:23.048811+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:04:54.306726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010610+0000","last_change":"2024-07-26T06:18:21.990382+0000","last_active":"2024-07-26T06:18:23.010610+0000","last_peered":"2024-07-26T06:18:23.010610+0000","last_clean":"2024-07-26T06:18:23.010610+0000","last_became_active":"2024-07-26T06:18:21.989820+0000","last_became_peered":"2024-07-26T06:18:21.989820+0000","last_unstale":"2024-07-26T06:18:23.010610+0000","last_undegraded":"2024-07-26T06:18:23.010610+0000","last_fullsized":"2024-07-26T06:18:23.010610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:41:36.941404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048466+0000","last_change":"2024-07-26T06:18:20.992668+0000","last_active":"2024-07-26T06:18:23.048466+0000","last_peered":"2024-07-26T06:18:23.048466+0000","last_clean":"2024-07-26T06:18:23.048466+0000","last_became_active":"2024-07-26T06:18:20.992363+0000","last_became_peered":"2024-07-26T06:18:20.992363+0000","last_unstale":"2024-07-26T06:18:23.048466+0000","last_undegraded":"2024-07-26T06:18:23.048466+0000","last_fullsized":"2024-07-26T06:18:23.048466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:10.468621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002701+0000","last_change":"2024-07-26T06:18:21.997699+0000","last_active":"2024-07-26T06:18:23.002701+0000","last_peered":"2024-07-26T06:18:23.002701+0000","last_clean":"2024-07-26T06:18:23.002701+0000","last_became_active":"2024-07-26T06:18:21.997385+0000","last_became_peered":"2024-07-26T06:18:21.997385+0000","last_unstale":"2024-07-26T06:18:23.002701+0000","last_undegraded":"2024-07-26T06:18:23.002701+0000","last_fullsized":"2024-07-26T06:18:23.002701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:36.955469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047913+0000","last_change":"2024-07-26T06:18:20.998822+0000","last_active":"2024-07-26T06:18:23.047913+0000","last_peered":"2024-07-26T06:18:23.047913+0000","last_clean":"2024-07-26T06:18:23.047913+0000","last_became_active":"2024-07-26T06:18:20.997942+0000","last_became_peered":"2024-07-26T06:18:20.997942+0000","last_unstale":"2024-07-26T06:18:23.047913+0000","last_undegraded":"2024-07-26T06:18:23.047913+0000","last_fullsized":"2024-07-26T06:18:23.047913+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:56:29.641105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010352+0000","last_change":"2024-07-26T06:18:21.986944+0000","last_active":"2024-07-26T06:18:23.010352+0000","last_peered":"2024-07-26T06:18:23.010352+0000","last_clean":"2024-07-26T06:18:23.010352+0000","last_became_active":"2024-07-26T06:18:21.986052+0000","last_became_peered":"2024-07-26T06:18:21.986052+0000","last_unstale":"2024-07-26T06:18:23.010352+0000","last_undegraded":"2024-07-26T06:18:23.010352+0000","last_fullsized":"2024-07-26T06:18:23.010352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:23:12.657539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000814+0000","last_change":"2024-07-26T06:18:20.989516+0000","last_active":"2024-07-26T06:18:23.000814+0000","last_peered":"2024-07-26T06:18:23.000814+0000","last_clean":"2024-07-26T06:18:23.000814+0000","last_became_active":"2024-07-26T06:18:20.989233+0000","last_became_peered":"2024-07-26T06:18:20.989233+0000","last_unstale":"2024-07-26T06:18:23.000814+0000","last_undegraded":"2024-07-26T06:18:23.000814+0000","last_fullsized":"2024-07-26T06:18:23.000814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:41.387341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049589+0000","last_change":"2024-07-26T06:18:22.006076+0000","last_active":"2024-07-26T06:18:23.049589+0000","last_peered":"2024-07-26T06:18:23.049589+0000","last_clean":"2024-07-26T06:18:23.049589+0000","last_became_active":"2024-07-26T06:18:22.005777+0000","last_became_peered":"2024-07-26T06:18:22.005777+0000","last_unstale":"2024-07-26T06:18:23.049589+0000","last_undegraded":"2024-07-26T06:18:23.049589+0000","last_fullsized":"2024-07-26T06:18:23.049589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:25:59.016108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002647+0000","last_change":"2024-07-26T06:18:20.996330+0000","last_active":"2024-07-26T06:18:23.002647+0000","last_peered":"2024-07-26T06:18:23.002647+0000","last_clean":"2024-07-26T06:18:23.002647+0000","last_became_active":"2024-07-26T06:18:20.995995+0000","last_became_peered":"2024-07-26T06:18:20.995995+0000","last_unstale":"2024-07-26T06:18:23.002647+0000","last_undegraded":"2024-07-26T06:18:23.002647+0000","last_fullsized":"2024-07-26T06:18:23.002647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:03:39.536261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000396+0000","last_change":"2024-07-26T06:18:22.005673+0000","last_active":"2024-07-26T06:18:23.000396+0000","last_peered":"2024-07-26T06:18:23.000396+0000","last_clean":"2024-07-26T06:18:23.000396+0000","last_became_active":"2024-07-26T06:18:22.005367+0000","last_became_peered":"2024-07-26T06:18:22.005367+0000","last_unstale":"2024-07-26T06:18:23.000396+0000","last_undegraded":"2024-07-26T06:18:23.000396+0000","last_fullsized":"2024-07-26T06:18:23.000396+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:28:57.494896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046812+0000","last_change":"2024-07-26T06:18:20.999272+0000","last_active":"2024-07-26T06:18:23.046812+0000","last_peered":"2024-07-26T06:18:23.046812+0000","last_clean":"2024-07-26T06:18:23.046812+0000","last_became_active":"2024-07-26T06:18:20.998332+0000","last_became_peered":"2024-07-26T06:18:20.998332+0000","last_unstale":"2024-07-26T06:18:23.046812+0000","last_undegraded":"2024-07-26T06:18:23.046812+0000","last_fullsized":"2024-07-26T06:18:23.046812+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:04:54.306726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.000324+0000","last_change":"2024-07-26T06:18:22.000448+0000","last_active":"2024-07-26T06:18:23.000324+0000","last_peered":"2024-07-26T06:18:23.000324+0000","last_clean":"2024-07-26T06:18:23.000324+0000","last_became_active":"2024-07-26T06:18:21.998821+0000","last_became_peered":"2024-07-26T06:18:21.998821+0000","last_unstale":"2024-07-26T06:18:23.000324+0000","last_undegraded":"2024-07-26T06:18:23.000324+0000","last_fullsized":"2024-07-26T06:18:23.000324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:02:37.809369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010168+0000","last_change":"2024-07-26T06:18:20.989738+0000","last_active":"2024-07-26T06:18:23.010168+0000","last_peered":"2024-07-26T06:18:23.010168+0000","last_clean":"2024-07-26T06:18:23.010168+0000","last_became_active":"2024-07-26T06:18:20.989192+0000","last_became_peered":"2024-07-26T06:18:20.989192+0000","last_unstale":"2024-07-26T06:18:23.010168+0000","last_undegraded":"2024-07-26T06:18:23.010168+0000","last_fullsized":"2024-07-26T06:18:23.010168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:47:19.107770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049845+0000","last_change":"2024-07-26T06:18:22.010828+0000","last_active":"2024-07-26T06:18:23.049845+0000","last_peered":"2024-07-26T06:18:23.049845+0000","last_clean":"2024-07-26T06:18:23.049845+0000","last_became_active":"2024-07-26T06:18:22.010554+0000","last_became_peered":"2024-07-26T06:18:22.010554+0000","last_unstale":"2024-07-26T06:18:23.049845+0000","last_undegraded":"2024-07-26T06:18:23.049845+0000","last_fullsized":"2024-07-26T06:18:23.049845+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:32:12.907681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050679+0000","last_change":"2024-07-26T06:18:20.997000+0000","last_active":"2024-07-26T06:18:23.050679+0000","last_peered":"2024-07-26T06:18:23.050679+0000","last_clean":"2024-07-26T06:18:23.050679+0000","last_became_active":"2024-07-26T06:18:20.996702+0000","last_became_peered":"2024-07-26T06:18:20.996702+0000","last_unstale":"2024-07-26T06:18:23.050679+0000","last_undegraded":"2024-07-26T06:18:23.050679+0000","last_fullsized":"2024-07-26T06:18:23.050679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:09:00.822200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050274+0000","last_change":"2024-07-26T06:18:22.006864+0000","last_active":"2024-07-26T06:18:23.050274+0000","last_peered":"2024-07-26T06:18:23.050274+0000","last_clean":"2024-07-26T06:18:23.050274+0000","last_became_active":"2024-07-26T06:18:22.006560+0000","last_became_peered":"2024-07-26T06:18:22.006560+0000","last_unstale":"2024-07-26T06:18:23.050274+0000","last_undegraded":"2024-07-26T06:18:23.050274+0000","last_fullsized":"2024-07-26T06:18:23.050274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:36:11.068267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010882+0000","last_change":"2024-07-26T06:18:20.994994+0000","last_active":"2024-07-26T06:18:23.010882+0000","last_peered":"2024-07-26T06:18:23.010882+0000","last_clean":"2024-07-26T06:18:23.010882+0000","last_became_active":"2024-07-26T06:18:20.994680+0000","last_became_peered":"2024-07-26T06:18:20.994680+0000","last_unstale":"2024-07-26T06:18:23.010882+0000","last_undegraded":"2024-07-26T06:18:23.010882+0000","last_fullsized":"2024-07-26T06:18:23.010882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:09.243999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002995+0000","last_change":"2024-07-26T06:18:22.005904+0000","last_active":"2024-07-26T06:18:23.002995+0000","last_peered":"2024-07-26T06:18:23.002995+0000","last_clean":"2024-07-26T06:18:23.002995+0000","last_became_active":"2024-07-26T06:18:22.005568+0000","last_became_peered":"2024-07-26T06:18:22.005568+0000","last_unstale":"2024-07-26T06:18:23.002995+0000","last_undegraded":"2024-07-26T06:18:23.002995+0000","last_fullsized":"2024-07-26T06:18:23.002995+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:27:01.477063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010690+0000","last_change":"2024-07-26T06:18:20.999298+0000","last_active":"2024-07-26T06:18:23.010690+0000","last_peered":"2024-07-26T06:18:23.010690+0000","last_clean":"2024-07-26T06:18:23.010690+0000","last_became_active":"2024-07-26T06:18:20.999013+0000","last_became_peered":"2024-07-26T06:18:20.999013+0000","last_unstale":"2024-07-26T06:18:23.010690+0000","last_undegraded":"2024-07-26T06:18:23.010690+0000","last_fullsized":"2024-07-26T06:18:23.010690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:26.719145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045970+0000","last_change":"2024-07-26T06:18:22.004090+0000","last_active":"2024-07-26T06:18:23.045970+0000","last_peered":"2024-07-26T06:18:23.045970+0000","last_clean":"2024-07-26T06:18:23.045970+0000","last_became_active":"2024-07-26T06:18:22.003821+0000","last_became_peered":"2024-07-26T06:18:22.003821+0000","last_unstale":"2024-07-26T06:18:23.045970+0000","last_undegraded":"2024-07-26T06:18:23.045970+0000","last_fullsized":"2024-07-26T06:18:23.045970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:10.411071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.003198+0000","last_change":"2024-07-26T06:18:20.993260+0000","last_active":"2024-07-26T06:18:23.003198+0000","last_peered":"2024-07-26T06:18:23.003198+0000","last_clean":"2024-07-26T06:18:23.003198+0000","last_became_active":"2024-07-26T06:18:20.992202+0000","last_became_peered":"2024-07-26T06:18:20.992202+0000","last_unstale":"2024-07-26T06:18:23.003198+0000","last_undegraded":"2024-07-26T06:18:23.003198+0000","last_fullsized":"2024-07-26T06:18:23.003198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:07:50.838016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047389+0000","last_change":"2024-07-26T06:18:22.022184+0000","last_active":"2024-07-26T06:18:23.047389+0000","last_peered":"2024-07-26T06:18:23.047389+0000","last_clean":"2024-07-26T06:18:23.047389+0000","last_became_active":"2024-07-26T06:18:22.021007+0000","last_became_peered":"2024-07-26T06:18:22.021007+0000","last_unstale":"2024-07-26T06:18:23.047389+0000","last_undegraded":"2024-07-26T06:18:23.047389+0000","last_fullsized":"2024-07-26T06:18:23.047389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:52:05.787110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045964+0000","last_change":"2024-07-26T06:18:20.989199+0000","last_active":"2024-07-26T06:18:23.045964+0000","last_peered":"2024-07-26T06:18:23.045964+0000","last_clean":"2024-07-26T06:18:23.045964+0000","last_became_active":"2024-07-26T06:18:20.988925+0000","last_became_peered":"2024-07-26T06:18:20.988925+0000","last_unstale":"2024-07-26T06:18:23.045964+0000","last_undegraded":"2024-07-26T06:18:23.045964+0000","last_fullsized":"2024-07-26T06:18:23.045964+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:04:52.286362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045869+0000","last_change":"2024-07-26T06:18:22.005967+0000","last_active":"2024-07-26T06:18:23.045869+0000","last_peered":"2024-07-26T06:18:23.045869+0000","last_clean":"2024-07-26T06:18:23.045869+0000","last_became_active":"2024-07-26T06:18:22.005683+0000","last_became_peered":"2024-07-26T06:18:22.005683+0000","last_unstale":"2024-07-26T06:18:23.045869+0000","last_undegraded":"2024-07-26T06:18:23.045869+0000","last_fullsized":"2024-07-26T06:18:23.045869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:56:03.312579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.045399+0000","last_change":"2024-07-26T06:18:21.000416+0000","last_active":"2024-07-26T06:18:23.045399+0000","last_peered":"2024-07-26T06:18:23.045399+0000","last_clean":"2024-07-26T06:18:23.045399+0000","last_became_active":"2024-07-26T06:18:21.000094+0000","last_became_peered":"2024-07-26T06:18:21.000094+0000","last_unstale":"2024-07-26T06:18:23.045399+0000","last_undegraded":"2024-07-26T06:18:23.045399+0000","last_fullsized":"2024-07-26T06:18:23.045399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:20.661146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046621+0000","last_change":"2024-07-26T06:18:22.016408+0000","last_active":"2024-07-26T06:18:23.046621+0000","last_peered":"2024-07-26T06:18:23.046621+0000","last_clean":"2024-07-26T06:18:23.046621+0000","last_became_active":"2024-07-26T06:18:22.016084+0000","last_became_peered":"2024-07-26T06:18:22.016084+0000","last_unstale":"2024-07-26T06:18:23.046621+0000","last_undegraded":"2024-07-26T06:18:23.046621+0000","last_fullsized":"2024-07-26T06:18:23.046621+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:27.725636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002351+0000","last_change":"2024-07-26T06:18:20.993457+0000","last_active":"2024-07-26T06:18:23.002351+0000","last_peered":"2024-07-26T06:18:23.002351+0000","last_clean":"2024-07-26T06:18:23.002351+0000","last_became_active":"2024-07-26T06:18:20.991482+0000","last_became_peered":"2024-07-26T06:18:20.991482+0000","last_unstale":"2024-07-26T06:18:23.002351+0000","last_undegraded":"2024-07-26T06:18:23.002351+0000","last_fullsized":"2024-07-26T06:18:23.002351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:04.206969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002742+0000","last_change":"2024-07-26T06:18:22.006079+0000","last_active":"2024-07-26T06:18:23.002742+0000","last_peered":"2024-07-26T06:18:23.002742+0000","last_clean":"2024-07-26T06:18:23.002742+0000","last_became_active":"2024-07-26T06:18:22.005776+0000","last_became_peered":"2024-07-26T06:18:22.005776+0000","last_unstale":"2024-07-26T06:18:23.002742+0000","last_undegraded":"2024-07-26T06:18:23.002742+0000","last_fullsized":"2024-07-26T06:18:23.002742+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.347684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001579+0000","last_change":"2024-07-26T06:18:20.985608+0000","last_active":"2024-07-26T06:18:23.001579+0000","last_peered":"2024-07-26T06:18:23.001579+0000","last_clean":"2024-07-26T06:18:23.001579+0000","last_became_active":"2024-07-26T06:18:20.985339+0000","last_became_peered":"2024-07-26T06:18:20.985339+0000","last_unstale":"2024-07-26T06:18:23.001579+0000","last_undegraded":"2024-07-26T06:18:23.001579+0000","last_fullsized":"2024-07-26T06:18:23.001579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:03:02.983465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.050080+0000","last_change":"2024-07-26T06:18:22.008331+0000","last_active":"2024-07-26T06:18:23.050080+0000","last_peered":"2024-07-26T06:18:23.050080+0000","last_clean":"2024-07-26T06:18:23.050080+0000","last_became_active":"2024-07-26T06:18:22.007887+0000","last_became_peered":"2024-07-26T06:18:22.007887+0000","last_unstale":"2024-07-26T06:18:23.050080+0000","last_undegraded":"2024-07-26T06:18:23.050080+0000","last_fullsized":"2024-07-26T06:18:23.050080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:10:24.277833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001515+0000","last_change":"2024-07-26T06:18:20.986447+0000","last_active":"2024-07-26T06:18:23.001515+0000","last_peered":"2024-07-26T06:18:23.001515+0000","last_clean":"2024-07-26T06:18:23.001515+0000","last_became_active":"2024-07-26T06:18:20.985976+0000","last_became_peered":"2024-07-26T06:18:20.985976+0000","last_unstale":"2024-07-26T06:18:23.001515+0000","last_undegraded":"2024-07-26T06:18:23.001515+0000","last_fullsized":"2024-07-26T06:18:23.001515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:56.859236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048688+0000","last_change":"2024-07-26T06:18:21.997149+0000","last_active":"2024-07-26T06:18:23.048688+0000","last_peered":"2024-07-26T06:18:23.048688+0000","last_clean":"2024-07-26T06:18:23.048688+0000","last_became_active":"2024-07-26T06:18:21.996234+0000","last_became_peered":"2024-07-26T06:18:21.996234+0000","last_unstale":"2024-07-26T06:18:23.048688+0000","last_undegraded":"2024-07-26T06:18:23.048688+0000","last_fullsized":"2024-07-26T06:18:23.048688+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:04:51.949797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048186+0000","last_change":"2024-07-26T06:18:20.995565+0000","last_active":"2024-07-26T06:18:23.048186+0000","last_peered":"2024-07-26T06:18:23.048186+0000","last_clean":"2024-07-26T06:18:23.048186+0000","last_became_active":"2024-07-26T06:18:20.994334+0000","last_became_peered":"2024-07-26T06:18:20.994334+0000","last_unstale":"2024-07-26T06:18:23.048186+0000","last_undegraded":"2024-07-26T06:18:23.048186+0000","last_fullsized":"2024-07-26T06:18:23.048186+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:15:14.727277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.003095+0000","last_change":"2024-07-26T06:18:21.986331+0000","last_active":"2024-07-26T06:18:23.003095+0000","last_peered":"2024-07-26T06:18:23.003095+0000","last_clean":"2024-07-26T06:18:23.003095+0000","last_became_active":"2024-07-26T06:18:21.985867+0000","last_became_peered":"2024-07-26T06:18:21.985867+0000","last_unstale":"2024-07-26T06:18:23.003095+0000","last_undegraded":"2024-07-26T06:18:23.003095+0000","last_fullsized":"2024-07-26T06:18:23.003095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:49.028809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.048861+0000","last_change":"2024-07-26T06:18:20.995274+0000","last_active":"2024-07-26T06:18:23.048861+0000","last_peered":"2024-07-26T06:18:23.048861+0000","last_clean":"2024-07-26T06:18:23.048861+0000","last_became_active":"2024-07-26T06:18:20.994117+0000","last_became_peered":"2024-07-26T06:18:20.994117+0000","last_unstale":"2024-07-26T06:18:23.048861+0000","last_undegraded":"2024-07-26T06:18:23.048861+0000","last_fullsized":"2024-07-26T06:18:23.048861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:56:29.641105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.046946+0000","last_change":"2024-07-26T06:18:22.009546+0000","last_active":"2024-07-26T06:18:23.046946+0000","last_peered":"2024-07-26T06:18:23.046946+0000","last_clean":"2024-07-26T06:18:23.046946+0000","last_became_active":"2024-07-26T06:18:22.008954+0000","last_became_peered":"2024-07-26T06:18:22.008954+0000","last_unstale":"2024-07-26T06:18:23.046946+0000","last_undegraded":"2024-07-26T06:18:23.046946+0000","last_fullsized":"2024-07-26T06:18:23.046946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:34:45.374115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010756+0000","last_change":"2024-07-26T06:18:20.991232+0000","last_active":"2024-07-26T06:18:23.010756+0000","last_peered":"2024-07-26T06:18:23.010756+0000","last_clean":"2024-07-26T06:18:23.010756+0000","last_became_active":"2024-07-26T06:18:20.990934+0000","last_became_peered":"2024-07-26T06:18:20.990934+0000","last_unstale":"2024-07-26T06:18:23.010756+0000","last_undegraded":"2024-07-26T06:18:23.010756+0000","last_fullsized":"2024-07-26T06:18:23.010756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:16:26.681319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.010577+0000","last_change":"2024-07-26T06:18:21.991776+0000","last_active":"2024-07-26T06:18:23.010577+0000","last_peered":"2024-07-26T06:18:23.010577+0000","last_clean":"2024-07-26T06:18:23.010577+0000","last_became_active":"2024-07-26T06:18:21.991435+0000","last_became_peered":"2024-07-26T06:18:21.991435+0000","last_unstale":"2024-07-26T06:18:23.010577+0000","last_undegraded":"2024-07-26T06:18:23.010577+0000","last_fullsized":"2024-07-26T06:18:23.010577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:58:51.096745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002245+0000","last_change":"2024-07-26T06:18:20.995362+0000","last_active":"2024-07-26T06:18:23.002245+0000","last_peered":"2024-07-26T06:18:23.002245+0000","last_clean":"2024-07-26T06:18:23.002245+0000","last_became_active":"2024-07-26T06:18:20.995037+0000","last_became_peered":"2024-07-26T06:18:20.995037+0000","last_unstale":"2024-07-26T06:18:23.002245+0000","last_undegraded":"2024-07-26T06:18:23.002245+0000","last_fullsized":"2024-07-26T06:18:23.002245+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:37:09.935711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049095+0000","last_change":"2024-07-26T06:18:22.003755+0000","last_active":"2024-07-26T06:18:23.049095+0000","last_peered":"2024-07-26T06:18:23.049095+0000","last_clean":"2024-07-26T06:18:23.049095+0000","last_became_active":"2024-07-26T06:18:22.003482+0000","last_became_peered":"2024-07-26T06:18:22.003482+0000","last_unstale":"2024-07-26T06:18:23.049095+0000","last_undegraded":"2024-07-26T06:18:23.049095+0000","last_fullsized":"2024-07-26T06:18:23.049095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:41:36.941404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047674+0000","last_change":"2024-07-26T06:18:20.998679+0000","last_active":"2024-07-26T06:18:23.047674+0000","last_peered":"2024-07-26T06:18:23.047674+0000","last_clean":"2024-07-26T06:18:23.047674+0000","last_became_active":"2024-07-26T06:18:20.995354+0000","last_became_peered":"2024-07-26T06:18:20.995354+0000","last_unstale":"2024-07-26T06:18:23.047674+0000","last_undegraded":"2024-07-26T06:18:23.047674+0000","last_fullsized":"2024-07-26T06:18:23.047674+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:18:00.461542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.047774+0000","last_change":"2024-07-26T06:18:21.996805+0000","last_active":"2024-07-26T06:18:23.047774+0000","last_peered":"2024-07-26T06:18:23.047774+0000","last_clean":"2024-07-26T06:18:23.047774+0000","last_became_active":"2024-07-26T06:18:21.996545+0000","last_became_peered":"2024-07-26T06:18:21.996545+0000","last_unstale":"2024-07-26T06:18:23.047774+0000","last_undegraded":"2024-07-26T06:18:23.047774+0000","last_fullsized":"2024-07-26T06:18:23.047774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:23:12.657539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.002045+0000","last_change":"2024-07-26T06:18:20.997537+0000","last_active":"2024-07-26T06:18:23.002045+0000","last_peered":"2024-07-26T06:18:23.002045+0000","last_clean":"2024-07-26T06:18:23.002045+0000","last_became_active":"2024-07-26T06:18:20.996601+0000","last_became_peered":"2024-07-26T06:18:20.996601+0000","last_unstale":"2024-07-26T06:18:23.002045+0000","last_undegraded":"2024-07-26T06:18:23.002045+0000","last_fullsized":"2024-07-26T06:18:23.002045+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:25:57.624539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.001239+0000","last_change":"2024-07-26T06:18:22.010740+0000","last_active":"2024-07-26T06:18:23.001239+0000","last_peered":"2024-07-26T06:18:23.001239+0000","last_clean":"2024-07-26T06:18:23.001239+0000","last_became_active":"2024-07-26T06:18:22.010400+0000","last_became_peered":"2024-07-26T06:18:22.010400+0000","last_unstale":"2024-07-26T06:18:23.001239+0000","last_undegraded":"2024-07-26T06:18:23.001239+0000","last_fullsized":"2024-07-26T06:18:23.001239+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:20.944385+0000","last_clean_scrub_stamp":"2024-07-26T06:18:20.944385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:25:58.631030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049813+0000","last_change":"2024-07-26T06:18:20.995328+0000","last_active":"2024-07-26T06:18:23.049813+0000","last_peered":"2024-07-26T06:18:23.049813+0000","last_clean":"2024-07-26T06:18:23.049813+0000","last_became_active":"2024-07-26T06:18:20.994643+0000","last_became_peered":"2024-07-26T06:18:20.994643+0000","last_unstale":"2024-07-26T06:18:23.049813+0000","last_undegraded":"2024-07-26T06:18:23.049813+0000","last_fullsized":"2024-07-26T06:18:23.049813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:19.937895+0000","last_clean_scrub_stamp":"2024-07-26T06:18:19.937895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:40.257374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"70'79","reported_seq":245,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-07-26T06:18:23.049232+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:18:23.049232+0000","last_peered":"2024-07-26T06:18:23.049232+0000","last_clean":"2024-07-26T06:18:23.049232+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:18:23.049232+0000","last_undegraded":"2024-07-26T06:18:23.049232+0000","last_fullsized":"2024-07-26T06:18:23.049232+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509504,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1585152,"data_stored":1012611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509505,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":999424,"data_stored":422867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509504,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1003520,"data_stored":423807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542208,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":422173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542207,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":423785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509506,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":422755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542209,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":422367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509505,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1585152,"data_stored":1012575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:18:26.887 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:18:26.887 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-07-26T06:18:27.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:18:27.045 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.139+0000 7f0303200640 1 -- 172.21.15.163:0/1111155099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040cee30 msgr2=0x7f0304150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.139+0000 7f0303200640 1 --2- 172.21.15.163:0/1111155099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040cee30 0x7f0304150ac0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005fd0 tx=0x7f02fc02ab30 comp rx=0 tx=0).stop 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.139+0000 7f0303200640 1 -- 172.21.15.163:0/1111155099 shutdown_connections 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.139+0000 7f0303200640 1 --2- 172.21.15.163:0/1111155099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040cee30 0x7f0304150ac0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.139+0000 7f0303200640 1 -- 172.21.15.163:0/1111155099 >> 172.21.15.163:0/1111155099 conn(0x7f03040cfa50 msgr2=0x7f03040cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.140+0000 7f0303200640 1 -- 172.21.15.163:0/1111155099 shutdown_connections 2024-07-26T06:18:27.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.140+0000 7f0303200640 1 -- 172.21.15.163:0/1111155099 wait complete. 2024-07-26T06:18:27.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.141+0000 7f0303200640 1 Processor -- start 2024-07-26T06:18:27.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.141+0000 7f0303200640 1 -- start start 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f0303200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03040ae790 0x7f03041f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f0303200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 0x7f03041f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f0303200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03040cee30 0x7f030415cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f0303200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0304151000 con 0x7f03040aeb70 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f0303200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f03040b3d80 con 0x7f03040ae790 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f0303200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0304151180 con 0x7f03040cee30 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 0x7f03041f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 0x7f03041f4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55298/0 (socket says 172.21.15.163:55298) 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.142+0000 7f02fbe00640 1 -- 172.21.15.163:0/690916523 learned_addr learned my addr 172.21.15.163:0/690916523 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f0302800640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03040cee30 0x7f030415cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f0301e00640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03040ae790 0x7f03041f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fbe00640 1 -- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03040cee30 msgr2=0x7f030415cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fbe00640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03040cee30 0x7f030415cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fbe00640 1 -- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03040ae790 msgr2=0x7f03041f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fbe00640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03040ae790 0x7f03041f6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fbe00640 1 -- 172.21.15.163:0/690916523 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03041f7330 con 0x7f03040aeb70 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fbe00640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 0x7f03041f4a00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00c450 tx=0x7f02ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0052d0 con 0x7f03040aeb70 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02ec005470 con 0x7f03040aeb70 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03041f4f40 con 0x7f03040aeb70 2024-07-26T06:18:27.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec012870 con 0x7f03040aeb70 2024-07-26T06:18:27.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.143+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030415c100 con 0x7f03040aeb70 2024-07-26T06:18:27.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.145+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0304150ac0 con 0x7f03040aeb70 2024-07-26T06:18:27.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.146+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f02ec012a10 con 0x7f03040aeb70 2024-07-26T06:18:27.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.147+0000 7f02fa000640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f02e00763f0 0x7f02e00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.147+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f02ec0975e0 con 0x7f03040aeb70 2024-07-26T06:18:27.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.147+0000 7f0301e00640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f02e00763f0 0x7f02e00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.148+0000 7f0301e00640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f02e00763f0 0x7f02e00788b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02f4000f60 tx=0x7f02f4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:27.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.151+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f03041f4f40 con 0x7f03040aeb70 2024-07-26T06:18:27.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.430+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f03040ce500 con 0x7f03040aeb70 2024-07-26T06:18:27.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.435+0000 7f02fa000640 1 -- 172.21.15.163:0/690916523 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f02ec060d90 con 0x7f03040aeb70 2024-07-26T06:18:27.435 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:18:27.435 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:18:27.436 INFO:teuthology.orchestra.run.smithi163.stdout: caps mds = "allow" 2024-07-26T06:18:27.436 INFO:teuthology.orchestra.run.smithi163.stdout: caps mon = "allow r" 2024-07-26T06:18:27.436 INFO:teuthology.orchestra.run.smithi163.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-07-26T06:18:27.436 INFO:teuthology.orchestra.run.smithi163.stderr:updated caps for client.0 2024-07-26T06:18:27.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f02e00763f0 msgr2=0x7f02e00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f02e00763f0 0x7f02e00788b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02f4000f60 tx=0x7f02f4009040 comp rx=0 tx=0).stop 2024-07-26T06:18:27.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 msgr2=0x7f03041f4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 0x7f03041f4a00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00c450 tx=0x7f02ec00c910 comp rx=0 tx=0).stop 2024-07-26T06:18:27.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 shutdown_connections 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f02e00763f0 0x7f02e00788b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03040cee30 0x7f030415cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03040aeb70 0x7f03041f4a00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 --2- 172.21.15.163:0/690916523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03040ae790 0x7f03041f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 >> 172.21.15.163:0/690916523 conn(0x7f03040cfa50 msgr2=0x7f03040ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 shutdown_connections 2024-07-26T06:18:27.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.438+0000 7f0303200640 1 -- 172.21.15.163:0/690916523 wait complete. 2024-07-26T06:18:27.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.706+0000 7fca3d200640 1 -- 172.21.15.163:0/1069220882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca1c003a30 msgr2=0x7fca1c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.706+0000 7fca3d200640 1 --2- 172.21.15.163:0/1069220882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca1c003a30 0x7fca1c003e50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fca3806ceb0 tx=0x7fca1c005830 comp rx=0 tx=0).stop 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.706+0000 7fca37e00640 1 -- 172.21.15.163:0/1069220882 reap_dead start 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.707+0000 7fca3d200640 1 -- 172.21.15.163:0/1069220882 shutdown_connections 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.707+0000 7fca3d200640 1 -- 172.21.15.163:0/1069220882 >> 172.21.15.163:0/1069220882 conn(0x7fca380b0e30 msgr2=0x7fca380b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.707+0000 7fca3d200640 1 -- 172.21.15.163:0/1069220882 shutdown_connections 2024-07-26T06:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.707+0000 7fca3d200640 1 -- 172.21.15.163:0/1069220882 wait complete. 2024-07-26T06:18:27.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.708+0000 7fca3d200640 1 Processor -- start 2024-07-26T06:18:27.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.708+0000 7fca3d200640 1 -- start start 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.709+0000 7fca3d200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1c003a30 0x7fca380cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.709+0000 7fca3d200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 0x7fca380ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.709+0000 7fca3d200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca380ade50 0x7fca38149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.709+0000 7fca3d200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca380b3e20 con 0x7fca3814bd60 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.709+0000 7fca3d200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca380cdfe0 con 0x7fca380ade50 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.709+0000 7fca3d200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fca380b3fa0 con 0x7fca1c003a30 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 0x7fca380ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 0x7fca380ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55320/0 (socket says 172.21.15.163:55320) 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 -- 172.21.15.163:0/904108872 learned_addr learned my addr 172.21.15.163:0/904108872 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:27.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca37400640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1c003a30 0x7fca380cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 -- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1c003a30 msgr2=0x7fca380cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1c003a30 0x7fca380cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca37e00640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca380ade50 0x7fca38149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 -- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca380ade50 msgr2=0x7fca38149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca380ade50 0x7fca38149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 -- 172.21.15.163:0/904108872 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca380cd5a0 con 0x7fca3814bd60 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.710+0000 7fca2fe00640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 0x7fca380ad910 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fca2000d580 tx=0x7fca2000da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.711+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20016020 con 0x7fca3814bd60 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.711+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca20012630 con 0x7fca3814bd60 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.711+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20012a80 con 0x7fca3814bd60 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.711+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca380cbcb0 con 0x7fca3814bd60 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.711+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca380cc270 con 0x7fca3814bd60 2024-07-26T06:18:27.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.712+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca380b4500 con 0x7fca3814bd60 2024-07-26T06:18:27.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.713+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fca20019020 con 0x7fca3814bd60 2024-07-26T06:18:27.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.714+0000 7fca2f400640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fca14076420 0x7fca140788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:27.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.714+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fca200976c0 con 0x7fca3814bd60 2024-07-26T06:18:27.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.714+0000 7fca37400640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fca14076420 0x7fca140788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:27.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.715+0000 7fca37400640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fca14076420 0x7fca140788e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca28036000 tx=0x7fca280046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:27.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.718+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fca20020020 con 0x7fca3814bd60 2024-07-26T06:18:27.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.943+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca380ceaf0 con 0x7fca3814bd60 2024-07-26T06:18:27.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.944+0000 7fca2f400640 1 -- 172.21.15.163:0/904108872 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5234 (secure 0 0 0) 0x7fca20067e20 con 0x7fca3814bd60 2024-07-26T06:18:27.944 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:27.945 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":55,"btime":"2024-07-26T06:18:25:009001+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6504,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/2494990100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":2494990100},{"type":"v1","addr":"172.21.15.163:6834","nonce":2494990100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:21.433788+0000","modified":"2024-07-26T06:18:24.503309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6516":{"gid":6516,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6834/3348438490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3348438490},{"type":"v1","addr":"172.21.15.164:6834","nonce":3348438490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6516,"qdb_cluster":[6516]},"id":6}]} 2024-07-26T06:18:27.945 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 55 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fca14076420 msgr2=0x7fca140788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fca14076420 0x7fca140788e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca28036000 tx=0x7fca280046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 msgr2=0x7fca380ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 0x7fca380ad910 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fca2000d580 tx=0x7fca2000da40 comp rx=0 tx=0).stop 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 shutdown_connections 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fca14076420 0x7fca140788e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca380ade50 0x7fca38149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca3814bd60 0x7fca380ad910 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 --2- 172.21.15.163:0/904108872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1c003a30 0x7fca380cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:27.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 >> 172.21.15.163:0/904108872 conn(0x7fca380b0e30 msgr2=0x7fca380af980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:27.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.947+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 shutdown_connections 2024-07-26T06:18:27.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:27.948+0000 7fca3d200640 1 -- 172.21.15.163:0/904108872 wait complete. 2024-07-26T06:18:28.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:18:21.433788+0000', 'modified': '2024-07-26T06:18:24.503309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6516}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6516': {'gid': 6516, 'name': 'b', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6834/3348438490', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6816', 'nonce': 3348438490}, {'type': 'v1', 'addr': '172.21.15.164:6834', 'nonce': 3348438490}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6516, 'qdb_cluster': [6516]} 2024-07-26T06:18:28.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:18:28.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-07-26T06:18:28.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.587+0000 7fc825c00640 1 -- 172.21.15.163:0/1310697912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 msgr2=0x7fc8040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.587+0000 7fc825c00640 1 --2- 172.21.15.163:0/1310697912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8040051c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fc818009ed0 tx=0x7fc804005830 comp rx=0 tx=0).stop 2024-07-26T06:18:28.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.587+0000 7fc825200640 1 -- 172.21.15.163:0/1310697912 reap_dead start 2024-07-26T06:18:28.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.587+0000 7fc825c00640 1 -- 172.21.15.163:0/1310697912 shutdown_connections 2024-07-26T06:18:28.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.587+0000 7fc825c00640 1 -- 172.21.15.163:0/1310697912 >> 172.21.15.163:0/1310697912 conn(0x7fc8200b0e30 msgr2=0x7fc8200b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.588+0000 7fc825c00640 1 -- 172.21.15.163:0/1310697912 shutdown_connections 2024-07-26T06:18:28.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.588+0000 7fc825c00640 1 -- 172.21.15.163:0/1310697912 wait complete. 2024-07-26T06:18:28.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.589+0000 7fc825c00640 1 Processor -- start 2024-07-26T06:18:28.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.589+0000 7fc825c00640 1 -- start start 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.590+0000 7fc825c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8201f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.590+0000 7fc825c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc82014faf0 0x7fc82015a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.590+0000 7fc825c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc820153210 0x7fc820157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.590+0000 7fc825c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc8200b3aa0 con 0x7fc804004d40 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.590+0000 7fc825c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc8200b3920 con 0x7fc82014faf0 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.590+0000 7fc825c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc8200b3c20 con 0x7fc820153210 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8201f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8201f7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55342/0 (socket says 172.21.15.163:55342) 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 -- 172.21.15.163:0/1686759856 learned_addr learned my addr 172.21.15.163:0/1686759856 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc817400640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc82014faf0 0x7fc82015a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 -- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc820153210 msgr2=0x7fc820157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc820153210 0x7fc820157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 -- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc82014faf0 msgr2=0x7fc82015a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc82014faf0 0x7fc82015a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 -- 172.21.15.163:0/1686759856 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8201546f0 con 0x7fc804004d40 2024-07-26T06:18:28.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.591+0000 7fc81fe00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8201f7e50 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fc810010950 tx=0x7fc810010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.592+0000 7fc81cc00640 1 -- 172.21.15.163:0/1686759856 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81000eda0 con 0x7fc804004d40 2024-07-26T06:18:28.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.592+0000 7fc81cc00640 1 -- 172.21.15.163:0/1686759856 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc810004030 con 0x7fc804004d40 2024-07-26T06:18:28.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.592+0000 7fc81cc00640 1 -- 172.21.15.163:0/1686759856 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810004460 con 0x7fc804004d40 2024-07-26T06:18:28.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.592+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8201549d0 con 0x7fc804004d40 2024-07-26T06:18:28.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.592+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc820159dc0 con 0x7fc804004d40 2024-07-26T06:18:28.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.594+0000 7fc81cc00640 1 -- 172.21.15.163:0/1686759856 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fc8100035f0 con 0x7fc804004d40 2024-07-26T06:18:28.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.596+0000 7fc81cc00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc7fc076420 0x7fc7fc0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.596+0000 7fc829fb8740 1 Processor -- start 2024-07-26T06:18:28.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.596+0000 7fc81cc00640 1 -- 172.21.15.163:0/1686759856 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc810098830 con 0x7fc804004d40 2024-07-26T06:18:28.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.596+0000 7fc817400640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc7fc076420 0x7fc7fc0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.596+0000 7fc829fb8740 1 -- start start 2024-07-26T06:18:28.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc817400640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc7fc076420 0x7fc7fc0788e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8201543f0 tx=0x7fc808001c30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 0x5564855f2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f18d0 0x5564855efea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ace50 0x5564855a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc829fb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5564854c23f0 con 0x5564855f2560 2024-07-26T06:18:28.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55648551d5c0 con 0x5564855ace50 2024-07-26T06:18:28.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.597+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55648550cb60 con 0x5564855f18d0 2024-07-26T06:18:28.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 0x5564855f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 0x5564855f2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55348/0 (socket says 172.21.15.163:55348) 2024-07-26T06:18:28.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 -- 172.21.15.163:0/3951582165 learned_addr learned my addr 172.21.15.163:0/3951582165 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:28.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 -- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f18d0 msgr2=0x5564855efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc825200640 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ace50 0x5564855a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f18d0 0x5564855efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 -- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ace50 msgr2=0x5564855a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ace50 0x5564855a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 -- 172.21.15.163:0/3951582165 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556485271850 con 0x5564855f2560 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc81fe00640 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 0x5564855f2940 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc81005d850 tx=0x7fc810060f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c10802b0e5d790e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.598+0000 7fc814c00640 1 -- 172.21.15.163:0/3951582165 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8100617d0 con 0x5564855f2560 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc814c00640 1 -- 172.21.15.163:0/3951582165 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc810065b80 con 0x5564855f2560 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc814c00640 1 -- 172.21.15.163:0/3951582165 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81006a5a0 con 0x5564855f2560 2024-07-26T06:18:28.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 -- 172.21.15.163:0/3951582165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 msgr2=0x5564855f2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 0x5564855f2940 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc81005d850 tx=0x7fc810060f40 comp rx=0 tx=0).stop 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 -- 172.21.15.163:0/3951582165 shutdown_connections 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ace50 0x5564855a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f18d0 0x5564855efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 --2- 172.21.15.163:0/3951582165 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855f2560 0x5564855f2940 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 -- 172.21.15.163:0/3951582165 >> 172.21.15.163:0/3951582165 conn(0x5564855a81a0 msgr2=0x5564855f05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 -- 172.21.15.163:0/3951582165 shutdown_connections 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.599+0000 7fc829fb8740 1 -- 172.21.15.163:0/3951582165 wait complete. 2024-07-26T06:18:28.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.600+0000 7fc829fb8740 1 -- start start 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.600+0000 7fc829fb8740 1 Processor -- start 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 0x5564855cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855cd2e0 0x5564855cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5564854b5210 con 0x5564855ca660 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556485510ba0 con 0x5564855f2560 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5564855f0ce0 con 0x5564855cd2e0 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.601+0000 7fc829fb8740 10 client.0 _ll_register_callbacks cb 0x5564854b58a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:18:28.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 0x5564855cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 0x5564855cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55354/0 (socket says 172.21.15.163:55354) 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 -- 172.21.15.163:0/694023127 learned_addr learned my addr 172.21.15.163:0/694023127 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc81fe00640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc825200640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855cd2e0 0x5564855cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 -- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855cd2e0 msgr2=0x5564855cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855cd2e0 0x5564855cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 -- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 msgr2=0x5564855c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 -- 172.21.15.163:0/694023127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564852e86b0 con 0x5564855ca660 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc817400640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 0x5564855cb4a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc8080146c0 tx=0x7fc808001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a6659c5219e4a1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.602+0000 7fc80d600640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc80d600640 1 -- 172.21.15.163:0/694023127 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808005320 con 0x5564855ca660 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc80d600640 1 -- 172.21.15.163:0/694023127 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc808022ad0 con 0x5564855ca660 2024-07-26T06:18:28.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55648551f510 con 0x5564855ca660 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc80d600640 1 -- 172.21.15.163:0/694023127 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fc808022cd0 con 0x5564855ca660 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 10 client.6639 fetch_fsmap learned FSMap version 55 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) -- 0x5564855cc0f0 con 0x5564855ca660 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc80d600640 1 -- 172.21.15.163:0/694023127 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55648551f510 con 0x5564855ca660 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc80d600640 1 -- 172.21.15.163:0/694023127 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 55) ==== 3502+0+0 (secure 0 0 0) 0x7fc808029050 con 0x5564855ca660 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 10 client.6639 fetch_fsmap finished waiting for FSMap version 55 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 10 client.6639 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 20 client.6639 populate_metadata read hostname 'smithi163' 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d2240 0x55648560e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 4 client.6639 mds_command: new command op to 6516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.603+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 --> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55648551afb0 con 0x5564855d2240 2024-07-26T06:18:28.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.604+0000 7fc81fe00640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d2240 0x55648560e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.605+0000 7fc81fe00640 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d2240 0x55648560e920 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2745b7e30740fc33 server_cookie=f4149c0519d2472 in_seq=0 out_seq=0 2024-07-26T06:18:28.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.605+0000 7fc80d600640 10 client.6639 ms_handle_connect on v2:172.21.15.164:6816/3348438490 2024-07-26T06:18:28.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.608+0000 7fc80d600640 1 -- 172.21.15.163:0/694023127 <== mds.0 v2:172.21.15.164:6816/3348438490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55648551afb0 con 0x5564855d2240 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.608+0000 7fc80d600640 10 client.6639 handle_command_reply: tid=0 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.608+0000 7fc829fb8740 1 client.6639 shutdown 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 msgr2=0x5564855cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 0x5564855cb4a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc8080146c0 tx=0x7fc808001a30 comp rx=0 tx=0).stop 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 shutdown_connections 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d2240 0x55648560e920 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855cd2e0 0x5564855cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855ca660 0x5564855cb4a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc80d600640 0 client.6639 ms_handle_reset on v2:172.21.15.164:6816/3348438490 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 --2- 172.21.15.163:0/694023127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 >> 172.21.15.163:0/694023127 conn(0x5564855ace50 msgr2=0x5564855abd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 shutdown_connections 2024-07-26T06:18:28.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 1 -- 172.21.15.163:0/694023127 wait complete. 2024-07-26T06:18:28.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.609+0000 7fc829fb8740 20 client.6639 trim_cache size 0 max 16384 2024-07-26T06:18:28.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.630+0000 7fc829fb8740 1 Processor -- start 2024-07-26T06:18:28.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.630+0000 7fc829fb8740 1 -- start start 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.630+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855ca660 0x5564855ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.630+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855ed4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855ab810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc829fb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc820200ba0 con 0x5564855cd2e0 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc8201ffc80 con 0x5564855f2560 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc8201f4750 con 0x5564855ca660 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855ed4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855ed4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55356/0 (socket says 172.21.15.163:55356) 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 -- 172.21.15.163:0/4099900525 learned_addr learned my addr 172.21.15.163:0/4099900525 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 -- 172.21.15.163:0/4099900525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855ca660 msgr2=0x5564855ecf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855ca660 0x5564855ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 -- 172.21.15.163:0/4099900525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 msgr2=0x5564855ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 -- 172.21.15.163:0/4099900525 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564855ffdb0 con 0x5564855cd2e0 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.631+0000 7fc817400640 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855ed4b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc808001df0 tx=0x7fc80800e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e235fba5038892a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc80fe00640 1 -- 172.21.15.163:0/4099900525 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808030030 con 0x5564855cd2e0 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc80fe00640 1 -- 172.21.15.163:0/4099900525 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc808003750 con 0x5564855cd2e0 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc80fe00640 1 -- 172.21.15.163:0/4099900525 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808003b80 con 0x5564855cd2e0 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 -- 172.21.15.163:0/4099900525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 msgr2=0x5564855ed4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855ed4b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc808001df0 tx=0x7fc80800e5a0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 -- 172.21.15.163:0/4099900525 shutdown_connections 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855f2560 0x5564855ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855ed4b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 --2- 172.21.15.163:0/4099900525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855ca660 0x5564855ecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.632+0000 7fc829fb8740 1 -- 172.21.15.163:0/4099900525 >> 172.21.15.163:0/4099900525 conn(0x5564855ace50 msgr2=0x5564855cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.633+0000 7fc829fb8740 1 -- 172.21.15.163:0/4099900525 shutdown_connections 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.633+0000 7fc829fb8740 1 -- 172.21.15.163:0/4099900525 wait complete. 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.633+0000 7fc829fb8740 1 -- start start 2024-07-26T06:18:28.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.634+0000 7fc829fb8740 1 Processor -- start 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ca660 0x55648566d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855caa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f2560 0x5564855abd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5564855cc990 con 0x5564855cd2e0 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc8201f4d70 con 0x5564855ca660 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x5564855d1c40 con 0x5564855f2560 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc829fb8740 10 client.0 _ll_register_callbacks cb 0x7fc80800ed70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55360/0 (socket says 172.21.15.163:55360) 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 -- 172.21.15.163:0/247751546 learned_addr learned my addr 172.21.15.163:0/247751546 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:28.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc81fe00640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ca660 0x55648566d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc825200640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f2560 0x5564855abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 -- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f2560 msgr2=0x5564855abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f2560 0x5564855abd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 -- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ca660 msgr2=0x55648566d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ca660 0x55648566d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.635+0000 7fc817400640 1 -- 172.21.15.163:0/247751546 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564855d42a0 con 0x5564855cd2e0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc817400640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855caa40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc8080146c0 tx=0x7fc808022430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c768735e1aab5a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc80d600640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc80d600640 1 -- 172.21.15.163:0/247751546 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808022af0 con 0x5564855cd2e0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc80d600640 1 -- 172.21.15.163:0/247751546 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fc80802b470 con 0x5564855cd2e0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc808003750 con 0x5564855cd2e0 2024-07-26T06:18:28.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc80d600640 1 -- 172.21.15.163:0/247751546 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fc808042050 con 0x5564855cd2e0 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc829fb8740 10 client.6645 fetch_fsmap learned FSMap version 55 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.636+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) -- 0x5564855ab880 con 0x5564855cd2e0 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc80d600640 1 -- 172.21.15.163:0/247751546 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc808003750 con 0x5564855cd2e0 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc80d600640 1 -- 172.21.15.163:0/247751546 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 55) ==== 3502+0+0 (secure 0 0 0) 0x7fc808045020 con 0x5564855cd2e0 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc829fb8740 10 client.6645 fetch_fsmap finished waiting for FSMap version 55 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc829fb8740 10 client.6645 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc829fb8740 20 client.6645 populate_metadata read hostname 'smithi163' 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc829fb8740 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d08c0 0x5564855f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc829fb8740 4 client.6645 mds_command: new command op to 6516 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:18:28.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 --> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] -- command(tid 0: {"prefix": "status"}) -- 0x5564855d3810 con 0x5564855d08c0 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.637+0000 7fc81fe00640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d08c0 0x5564855f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.638+0000 7fc81fe00640 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d08c0 0x5564855f39d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=334eb4e31b512a8b server_cookie=b0630bd5abb26e55 in_seq=0 out_seq=0 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.638+0000 7fc80d600640 10 client.6645 ms_handle_connect on v2:172.21.15.164:6816/3348438490 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc80d600640 1 -- 172.21.15.163:0/247751546 <== mds.0 v2:172.21.15.164:6816/3348438490 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5564855d3810 con 0x5564855d08c0 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc80d600640 10 client.6645 handle_command_reply: tid=0 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 client.6645 shutdown 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 msgr2=0x5564855caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855caa40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc8080146c0 tx=0x7fc808022430 comp rx=0 tx=0).stop 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 shutdown_connections 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:6816/3348438490,v1:172.21.15.164:6834/3348438490] conn(0x5564855d08c0 0x5564855f39d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5564855f2560 0x5564855abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5564855cd2e0 0x5564855caa40 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 --2- 172.21.15.163:0/247751546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5564855ca660 0x55648566d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 >> 172.21.15.163:0/247751546 conn(0x5564855ace50 msgr2=0x5564855ecf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc80d600640 0 client.6645 ms_handle_reset on v2:172.21.15.164:6816/3348438490 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.639+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 shutdown_connections 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc829fb8740 1 -- 172.21.15.163:0/247751546 wait complete. 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc829fb8740 20 client.6645 trim_cache size 0 max 16384 2024-07-26T06:18:28.642 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 6516, 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 6.6480927230000004, 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 55, 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 81, 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 81, 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 11.654754579 2024-07-26T06:18:28.643 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:18:28.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc7fc076420 msgr2=0x7fc7fc0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc7fc076420 0x7fc7fc0788e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8201543f0 tx=0x7fc808001c30 comp rx=0 tx=0).stop 2024-07-26T06:18:28.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 msgr2=0x7fc8201f7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8201f7e50 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fc810010950 tx=0x7fc810010e10 comp rx=0 tx=0).stop 2024-07-26T06:18:28.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 shutdown_connections 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc7fc076420 0x7fc7fc0788e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc820153210 0x7fc820157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc82014faf0 0x7fc82015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 --2- 172.21.15.163:0/1686759856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc804004d40 0x7fc8201f7e50 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.640+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 >> 172.21.15.163:0/1686759856 conn(0x7fc8200b0e30 msgr2=0x7fc8200b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.641+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 shutdown_connections 2024-07-26T06:18:28.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.641+0000 7fc825c00640 1 -- 172.21.15.163:0/1686759856 wait complete. 2024-07-26T06:18:28.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:18:28.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71df200640 1 -- 172.21.15.163:0/2477481847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 msgr2=0x7f71c4005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71df200640 1 --2- 172.21.15.163:0/2477481847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71c4005d90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f71e00b1bc0 tx=0x7f71c4006940 comp rx=0 tx=0).stop 2024-07-26T06:18:28.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71de800640 1 -- 172.21.15.163:0/2477481847 reap_dead start 2024-07-26T06:18:28.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71df200640 1 -- 172.21.15.163:0/2477481847 shutdown_connections 2024-07-26T06:18:28.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71df200640 1 -- 172.21.15.163:0/2477481847 >> 172.21.15.163:0/2477481847 conn(0x7f71e00b0e30 msgr2=0x7f71e00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:28.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71df200640 1 -- 172.21.15.163:0/2477481847 shutdown_connections 2024-07-26T06:18:28.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.856+0000 7f71df200640 1 -- 172.21.15.163:0/2477481847 wait complete. 2024-07-26T06:18:28.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.858+0000 7f71df200640 1 Processor -- start 2024-07-26T06:18:28.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.858+0000 7f71df200640 1 -- start start 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71df200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71e00741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71df200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e0154e00 0x7f71e01e5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71df200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e0071860 0x7f71e0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71df200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f71e01594f0 con 0x7f71c4005910 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71df200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f71e0159370 con 0x7f71e0154e00 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71df200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f71e0159670 con 0x7f71e0071860 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71dde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71e00741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71dde00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71e00741e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55390/0 (socket says 172.21.15.163:55390) 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.859+0000 7f71dde00640 1 -- 172.21.15.163:0/1546813253 learned_addr learned my addr 172.21.15.163:0/1546813253 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71de800640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e0071860 0x7f71e0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71d7e00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e0154e00 0x7f71e01e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71dde00640 1 -- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e0071860 msgr2=0x7f71e0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71dde00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e0071860 0x7f71e0071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71dde00640 1 -- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e0154e00 msgr2=0x7f71e01e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71dde00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e0154e00 0x7f71e01e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71dde00640 1 -- 172.21.15.163:0/1546813253 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e0155c20 con 0x7f71c4005910 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71de800640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e0071860 0x7f71e0071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71d7e00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e0154e00 0x7f71e01e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:28.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71dde00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71e00741e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f71d0010a40 tx=0x7f71d0010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.860+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d000ed70 con 0x7f71c4005910 2024-07-26T06:18:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.861+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71d0004030 con 0x7f71c4005910 2024-07-26T06:18:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.861+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e0155f00 con 0x7f71c4005910 2024-07-26T06:18:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.861+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d0004440 con 0x7f71c4005910 2024-07-26T06:18:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.861+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e01564c0 con 0x7f71c4005910 2024-07-26T06:18:28.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.862+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e00cbad0 con 0x7f71c4005910 2024-07-26T06:18:28.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.863+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f71d00035f0 con 0x7f71c4005910 2024-07-26T06:18:28.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.864+0000 7f71d6000640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f71bc076420 0x7f71bc0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:28.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.864+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f71d00988a0 con 0x7f71c4005910 2024-07-26T06:18:28.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.865+0000 7f71d7e00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f71bc076420 0x7f71bc0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:28.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.865+0000 7f71d7e00640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f71bc076420 0x7f71bc0788e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f71c8032000 tx=0x7f71c8001190 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:28.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:28.869+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f71d0061f40 con 0x7f71c4005910 2024-07-26T06:18:29.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.117+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f71e01e5570 con 0x7f71c4005910 2024-07-26T06:18:29.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.118+0000 7f71d6000640 1 -- 172.21.15.163:0/1546813253 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) ==== 53+0+73 (secure 0 0 0) 0x7f71d0061840 con 0x7f71c4005910 2024-07-26T06:18:29.118 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:18:29.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.120+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f71bc076420 msgr2=0x7f71bc0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:29.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.120+0000 7f71df200640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f71bc076420 0x7f71bc0788e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f71c8032000 tx=0x7f71c8001190 comp rx=0 tx=0).stop 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.120+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 msgr2=0x7f71e00741e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.120+0000 7f71df200640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71e00741e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f71d0010a40 tx=0x7f71d0010f00 comp rx=0 tx=0).stop 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 shutdown_connections 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f71bc076420 0x7f71bc0788e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e0071860 0x7f71e0071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e0154e00 0x7f71e01e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 --2- 172.21.15.163:0/1546813253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71c4005910 0x7f71e00741e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 >> 172.21.15.163:0/1546813253 conn(0x7f71e00b0e30 msgr2=0x7f71e00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 shutdown_connections 2024-07-26T06:18:29.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29.121+0000 7f71df200640 1 -- 172.21.15.163:0/1546813253 wait complete. 2024-07-26T06:18:29.138 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:18:29.138 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:18:29.138 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:18:29.138 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:18:29.138 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:18:29.139 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:18:29.139 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:18:29.139 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:18:29.139 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:18:29.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.139 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.186 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:18:29.187 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42296sec preferred_lft 42296sec 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:18:29.188 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:18:29.189 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether c2:4f:6d:14:5d:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:18:29.190 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::c04f:6dff:fe14:5d96/64 scope link 2024-07-26T06:18:29.190 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:29.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.191 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:18:29.310 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:18:29.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.315 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:18:29.367 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:18:29.367 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:18:29.367 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:18:29.367 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:18:29.367 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:18:29.367 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:18:29.367 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:18:29.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:18:29.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:18:29.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:18:29.368 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:18:29.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:18:29.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:18:29.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.582 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:18:29.631 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:18:29.632 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:18:29.632 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:18:29.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:18:29.713 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:18:29.713 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:18:29.764 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:18:29.765 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:18:29.765 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:18:29.817 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:18:29.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.817 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:18:29.936 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:18:29.937 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:18:29.937 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:18:29.960 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:18:29.960 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:18:29.960 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:18:29.961 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:18:29.961 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:18:29.987 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:18:29.988 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:18:29.990 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:18:29.990 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:18:29.990 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:18:29.990 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:18:29.990 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:18:29.994 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:18:29.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:29.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:18:30.070 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:18:30.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:18:30.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:18:30.208 INFO:teuthology.orchestra.run.smithi163.stdout:client5533 2024-07-26T06:18:30.209 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:30.209 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:30.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 -- 172.21.15.163:0/1793932641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf8146b00 msgr2=0x7fcdf8161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 --2- 172.21.15.163:0/1793932641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf8146b00 0x7fcdf8161340 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8069110 tx=0x7fcdf002aa30 comp rx=0 tx=0).stop 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 -- 172.21.15.163:0/1793932641 shutdown_connections 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 --2- 172.21.15.163:0/1793932641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf8146b00 0x7fcdf8161340 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 --2- 172.21.15.163:0/1793932641 >> v2:172.21.15.164:3300/0 conn(0x7fcdf81465b0 0x7fcdf8160e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 --2- 172.21.15.163:0/1793932641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdf806bf40 0x7fcdf806c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 -- 172.21.15.163:0/1793932641 >> 172.21.15.163:0/1793932641 conn(0x7fcdf80b1ee0 msgr2=0x7fcdf80b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:30.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 -- 172.21.15.163:0/1793932641 shutdown_connections 2024-07-26T06:18:30.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.511+0000 7fcdfe800640 1 -- 172.21.15.163:0/1793932641 wait complete. 2024-07-26T06:18:30.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.513+0000 7fcdfe800640 1 Processor -- start 2024-07-26T06:18:30.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.513+0000 7fcdfe800640 1 -- start start 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfe800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdf806bf40 0x7fcdf815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfe800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 0x7fcdf80c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfe800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdf8146b00 0x7fcdf815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfe800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf80724d0 con 0x7fcdf81465b0 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfe800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf8070a90 con 0x7fcdf8146b00 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfe800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf8161880 con 0x7fcdf806bf40 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 0x7fcdf80c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 0x7fcdf80c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55410/0 (socket says 172.21.15.163:55410) 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfca00640 1 -- 172.21.15.163:0/3528183905 learned_addr learned my addr 172.21.15.163:0/3528183905 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:30.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfd400640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdf806bf40 0x7fcdf815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfde00640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdf8146b00 0x7fcdf815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.514+0000 7fcdfca00640 1 -- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdf806bf40 msgr2=0x7fcdf815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfca00640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdf806bf40 0x7fcdf815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfca00640 1 -- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdf8146b00 msgr2=0x7fcdf815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfca00640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdf8146b00 0x7fcdf815fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfca00640 1 -- 172.21.15.163:0/3528183905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf80c67c0 con 0x7fcdf81465b0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfca00640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 0x7fcdf80c5ee0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d740 tx=0x7fcde400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4017070 con 0x7fcdf81465b0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcde400de50 con 0x7fcdf81465b0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4004020 con 0x7fcdf81465b0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf81f7350 con 0x7fcdf81465b0 2024-07-26T06:18:30.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.515+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf815d240 con 0x7fcdf81465b0 2024-07-26T06:18:30.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.516+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf806c320 con 0x7fcdf81465b0 2024-07-26T06:18:30.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.518+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fcde40041c0 con 0x7fcdf81465b0 2024-07-26T06:18:30.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.519+0000 7fcdf5600640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcdd80763f0 0x7fcdd80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:30.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.519+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fcde4097a80 con 0x7fcdf81465b0 2024-07-26T06:18:30.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.519+0000 7fcdfd400640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcdd80763f0 0x7fcdd80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:30.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.519+0000 7fcdfd400640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcdd80763f0 0x7fcdd80788b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8144140 tx=0x7fcdec005e50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:30.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.525+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcde40611c0 con 0x7fcdf81465b0 2024-07-26T06:18:30.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.773+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcdf80cfea0 con 0x7fcdf81465b0 2024-07-26T06:18:30.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.774+0000 7fcdf5600640 1 -- 172.21.15.163:0/3528183905 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) ==== 53+0+73 (secure 0 0 0) 0x7fcde4060ac0 con 0x7fcdf81465b0 2024-07-26T06:18:30.775 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcdd80763f0 msgr2=0x7fcdd80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcdd80763f0 0x7fcdd80788b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8144140 tx=0x7fcdec005e50 comp rx=0 tx=0).stop 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 msgr2=0x7fcdf80c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 0x7fcdf80c5ee0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d740 tx=0x7fcde400dc00 comp rx=0 tx=0).stop 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 shutdown_connections 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fcdd80763f0 0x7fcdd80788b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdf8146b00 0x7fcdf815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf81465b0 0x7fcdf80c5ee0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 --2- 172.21.15.163:0/3528183905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdf806bf40 0x7fcdf815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:30.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 >> 172.21.15.163:0/3528183905 conn(0x7fcdf80b1ee0 msgr2=0x7fcdf80ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:30.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 shutdown_connections 2024-07-26T06:18:30.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:30.777+0000 7fcdfe800640 1 -- 172.21.15.163:0/3528183905 wait complete. 2024-07-26T06:18:30.796 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:18:30.796 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:18:30.796 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:18:30.796 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:18:30.797 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:18:30.797 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:18:30.797 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:18:30.797 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:18:30.797 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:18:30.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:30.797 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:18:30.853 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:18:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:18:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42272sec preferred_lft 42272sec 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:18:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::58b1:99ff:fee0:4877/64 scope link 2024-07-26T06:18:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:18:30.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:30.857 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:18:30.934 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:18:30.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:30.938 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:18:30.990 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:18:30.990 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:18:30.990 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:18:30.990 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:18:30.990 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:18:30.991 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:18:30.991 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:18:30.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:18:30.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:18:30.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:18:30.991 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:18:31.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:18:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:18:31.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:18:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:18:31.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:31.199 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:18:31.246 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:18:31.247 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:18:31.248 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:18:31.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:31.248 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:18:31.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:18:31.330 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:18:31.381 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:18:31.381 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:18:31.381 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:18:31.430 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:18:31.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:31.430 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:18:31.553 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:18:31.554 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:18:31.554 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:18:31.575 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:18:31.575 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:18:31.576 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:18:31.576 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:18:31.576 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:18:31.602 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:18:31.605 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:18:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:18:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:18:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:18:31.605 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:18:31.609 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:18:31.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:31.609 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:18:31.675 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:18:31.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:18:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:18:31.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:18:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:18:31.790 INFO:teuthology.orchestra.run.smithi164.stdout:client6660 2024-07-26T06:18:31.790 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:31.790 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:31.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-07-26T06:18:32.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2fe00640 1 -- 172.21.15.163:0/124553173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 msgr2=0x7f8d1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/124553173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d1000b9c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800c610 tx=0x7f8d10005830 comp rx=0 tx=0).stop 2024-07-26T06:18:32.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2f400640 1 -- 172.21.15.163:0/124553173 reap_dead start 2024-07-26T06:18:32.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2fe00640 1 -- 172.21.15.163:0/124553173 shutdown_connections 2024-07-26T06:18:32.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2fe00640 1 -- 172.21.15.163:0/124553173 >> 172.21.15.163:0/124553173 conn(0x7f8d300cfda0 msgr2=0x7f8d300d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:32.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2fe00640 1 -- 172.21.15.163:0/124553173 shutdown_connections 2024-07-26T06:18:32.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.054+0000 7f8d2fe00640 1 -- 172.21.15.163:0/124553173 wait complete. 2024-07-26T06:18:32.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.056+0000 7f8d2fe00640 1 Processor -- start 2024-07-26T06:18:32.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.056+0000 7f8d2fe00640 1 -- start start 2024-07-26T06:18:32.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d301591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d301f33b0 0x7f8d300cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d300cb2d0 0x7f8d3015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8d300ce470 con 0x7f8d10004e60 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8d300ce2f0 con 0x7f8d300cb2d0 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8d300ce5f0 con 0x7f8d301f33b0 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d301591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d301591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55428/0 (socket says 172.21.15.163:55428) 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.057+0000 7f8d2ea00640 1 -- 172.21.15.163:0/1892892557 learned_addr learned my addr 172.21.15.163:0/1892892557 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2ea00640 1 -- 172.21.15.163:0/1892892557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d301f33b0 msgr2=0x7f8d300cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2ea00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d301f33b0 0x7f8d300cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2ea00640 1 -- 172.21.15.163:0/1892892557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d300cb2d0 msgr2=0x7f8d3015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2ea00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d300cb2d0 0x7f8d3015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2ea00640 1 -- 172.21.15.163:0/1892892557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d300cd330 con 0x7f8d10004e60 2024-07-26T06:18:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2ea00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d301591b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e990 tx=0x7f8d2000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000cb60 con 0x7f8d10004e60 2024-07-26T06:18:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d2000cdf0 con 0x7f8d10004e60 2024-07-26T06:18:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d300cd610 con 0x7f8d10004e60 2024-07-26T06:18:32.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.058+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200116d0 con 0x7f8d10004e60 2024-07-26T06:18:32.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.059+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d3015e960 con 0x7f8d10004e60 2024-07-26T06:18:32.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.060+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d300b64c0 con 0x7f8d10004e60 2024-07-26T06:18:32.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.061+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f8d20011870 con 0x7f8d10004e60 2024-07-26T06:18:32.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.062+0000 7f8d26a00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8d0c076420 0x7f8d0c0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.062+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f8d200984d0 con 0x7f8d10004e60 2024-07-26T06:18:32.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.062+0000 7f8d2e000640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8d0c076420 0x7f8d0c0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:32.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.063+0000 7f8d2e000640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8d0c076420 0x7f8d0c0788e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d3015c2f0 tx=0x7f8d180079b0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:32.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.066+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8d20016030 con 0x7f8d10004e60 2024-07-26T06:18:32.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.309+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f8d300d1cc0 con 0x7f8d10004e60 2024-07-26T06:18:32.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.364+0000 7f8d26a00640 1 -- 172.21.15.163:0/1892892557 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v56) ==== 136+0+0 (secure 0 0 0) 0x7f8d20061b70 con 0x7f8d10004e60 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8d0c076420 msgr2=0x7f8d0c0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8d0c076420 0x7f8d0c0788e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d3015c2f0 tx=0x7f8d180079b0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 msgr2=0x7f8d301591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d301591b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e990 tx=0x7f8d2000ee50 comp rx=0 tx=0).stop 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 shutdown_connections 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8d0c076420 0x7f8d0c0788e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d300cb2d0 0x7f8d3015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d301f33b0 0x7f8d300cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 --2- 172.21.15.163:0/1892892557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d10004e60 0x7f8d301591b0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 >> 172.21.15.163:0/1892892557 conn(0x7f8d300cfda0 msgr2=0x7f8d3015aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:32.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.367+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 shutdown_connections 2024-07-26T06:18:32.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.368+0000 7f8d2fe00640 1 -- 172.21.15.163:0/1892892557 wait complete. 2024-07-26T06:18:32.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-07-26T06:18:32.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.595+0000 7fb0d4a00640 1 -- 172.21.15.163:0/558499758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d00ce4a0 msgr2=0x7fb0d0071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.595+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/558499758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d00ce4a0 0x7fb0d0071530 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8005fd0 tx=0x7fb0b802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.595+0000 7fb0d4a00640 1 -- 172.21.15.163:0/558499758 shutdown_connections 2024-07-26T06:18:32.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.595+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/558499758 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d00ce4a0 0x7fb0d0071530 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.595+0000 7fb0d4a00640 1 -- 172.21.15.163:0/558499758 >> 172.21.15.163:0/558499758 conn(0x7fb0d00b0e30 msgr2=0x7fb0d00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:32.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.595+0000 7fb0d4a00640 1 -- 172.21.15.163:0/558499758 shutdown_connections 2024-07-26T06:18:32.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.596+0000 7fb0d4a00640 1 -- 172.21.15.163:0/558499758 wait complete. 2024-07-26T06:18:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.597+0000 7fb0d4a00640 1 Processor -- start 2024-07-26T06:18:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.597+0000 7fb0d4a00640 1 -- start start 2024-07-26T06:18:32.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.598+0000 7fb0d4a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0d0071a70 0x7fb0d014bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.598+0000 7fb0d4a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0d00ce4a0 0x7fb0d015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.598+0000 7fb0d4a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 0x7fb0d015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.598+0000 7fb0d4a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00b41f0 con 0x7fb0d01556f0 2024-07-26T06:18:32.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.598+0000 7fb0d4a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00b4070 con 0x7fb0d0071a70 2024-07-26T06:18:32.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.598+0000 7fb0d4a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d00b4370 con 0x7fb0d00ce4a0 2024-07-26T06:18:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 0x7fb0d015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 0x7fb0d015a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55444/0 (socket says 172.21.15.163:55444) 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 -- 172.21.15.163:0/2840309376 learned_addr learned my addr 172.21.15.163:0/2840309376 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0c5e00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0d00ce4a0 0x7fb0d015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0ce800640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0d0071a70 0x7fb0d014bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 -- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0d00ce4a0 msgr2=0x7fb0d015a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0d00ce4a0 0x7fb0d015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 -- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0d0071a70 msgr2=0x7fb0d014bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0d0071a70 0x7fb0d014bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 -- 172.21.15.163:0/2840309376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d00ccf60 con 0x7fb0d01556f0 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0ce800640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0d0071a70 0x7fb0d014bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.599+0000 7fb0cf200640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 0x7fb0d015a590 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c7a0 tx=0x7fb0c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.600+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8003b70 con 0x7fb0d01556f0 2024-07-26T06:18:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.600+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0c8003d10 con 0x7fb0d01556f0 2024-07-26T06:18:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.600+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c800aab0 con 0x7fb0d01556f0 2024-07-26T06:18:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.600+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d00cd240 con 0x7fb0d01556f0 2024-07-26T06:18:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.600+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d014b310 con 0x7fb0d01556f0 2024-07-26T06:18:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.601+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d0071530 con 0x7fb0d01556f0 2024-07-26T06:18:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.602+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb0c800ac50 con 0x7fb0d01556f0 2024-07-26T06:18:32.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.603+0000 7fb0c7400640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb0ac0763f0 0x7fb0ac0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:32.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.603+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb0c80977f0 con 0x7fb0d01556f0 2024-07-26T06:18:32.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.603+0000 7fb0ce800640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb0ac0763f0 0x7fb0ac0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:32.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.604+0000 7fb0ce800640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb0ac0763f0 0x7fb0ac0788b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000a810 tx=0x7fb0c0008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:32.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.608+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb0c8061030 con 0x7fb0d01556f0 2024-07-26T06:18:32.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.841+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fb0d00ced60 con 0x7fb0ac0763f0 2024-07-26T06:18:32.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.845+0000 7fb0c7400640 1 -- 172.21.15.163:0/2840309376 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fb0d00ced60 con 0x7fb0ac0763f0 2024-07-26T06:18:32.846 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:18:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephfs" 2024-07-26T06:18:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:32.847 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:18:32.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb0ac0763f0 msgr2=0x7fb0ac0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb0ac0763f0 0x7fb0ac0788b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000a810 tx=0x7fb0c0008040 comp rx=0 tx=0).stop 2024-07-26T06:18:32.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 msgr2=0x7fb0d015a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:32.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 0x7fb0d015a590 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c7a0 tx=0x7fb0c800cc60 comp rx=0 tx=0).stop 2024-07-26T06:18:32.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 shutdown_connections 2024-07-26T06:18:32.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb0ac0763f0 0x7fb0ac0788b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0d01556f0 0x7fb0d015a590 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0d00ce4a0 0x7fb0d015a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 --2- 172.21.15.163:0/2840309376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0d0071a70 0x7fb0d014bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:32.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 >> 172.21.15.163:0/2840309376 conn(0x7fb0d00b0e30 msgr2=0x7fb0d00cdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:32.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 shutdown_connections 2024-07-26T06:18:32.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:32.848+0000 7fb0d4a00640 1 -- 172.21.15.163:0/2840309376 wait complete. 2024-07-26T06:18:32.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:18:33.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 -- 172.21.15.163:0/4104820160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b00ce040 msgr2=0x7f88b00ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 --2- 172.21.15.163:0/4104820160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b00ce040 0x7f88b00ce4f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f88a8005fd0 tx=0x7f88a802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 -- 172.21.15.163:0/4104820160 shutdown_connections 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 --2- 172.21.15.163:0/4104820160 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b00ce040 0x7f88b00ce4f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 --2- 172.21.15.163:0/4104820160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b0070990 0x7f88b00b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 -- 172.21.15.163:0/4104820160 >> 172.21.15.163:0/4104820160 conn(0x7f88b00b3060 msgr2=0x7f88b00b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 -- 172.21.15.163:0/4104820160 shutdown_connections 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.066+0000 7f88afe00640 1 -- 172.21.15.163:0/4104820160 wait complete. 2024-07-26T06:18:33.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.068+0000 7f88afe00640 1 Processor -- start 2024-07-26T06:18:33.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.068+0000 7f88afe00640 1 -- start start 2024-07-26T06:18:33.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 0x7f88b00b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88afe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b0071990 0x7f88b00ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88afe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b00ce040 0x7f88b014ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88afe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f88b00cedc0 con 0x7f88b0070990 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88afe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f88b00cec40 con 0x7f88b0071990 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88afe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88b00cef40 con 0x7f88b00ce040 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88aea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 0x7f88b00b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88aea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 0x7f88b00b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55462/0 (socket says 172.21.15.163:55462) 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.069+0000 7f88aea00640 1 -- 172.21.15.163:0/510691586 learned_addr learned my addr 172.21.15.163:0/510691586 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88aea00640 1 -- 172.21.15.163:0/510691586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b00ce040 msgr2=0x7f88b014ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88aea00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b00ce040 0x7f88b014ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88aea00640 1 -- 172.21.15.163:0/510691586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b0071990 msgr2=0x7f88b00ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88aea00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b0071990 0x7f88b00ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88aea00640 1 -- 172.21.15.163:0/510691586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b00ae4a0 con 0x7f88b0070990 2024-07-26T06:18:33.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88aea00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 0x7f88b00b0a30 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c950 tx=0x7f88a000ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:33.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0007c10 con 0x7f88b0070990 2024-07-26T06:18:33.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88a000f450 con 0x7f88b0070990 2024-07-26T06:18:33.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a000f8a0 con 0x7f88b0070990 2024-07-26T06:18:33.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b00ae780 con 0x7f88b0070990 2024-07-26T06:18:33.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.070+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b0154730 con 0x7f88b0070990 2024-07-26T06:18:33.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.072+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b00b5e40 con 0x7f88b0070990 2024-07-26T06:18:33.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.073+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f88a000fa40 con 0x7f88b0070990 2024-07-26T06:18:33.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.074+0000 7f88a6a00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f888c0763f0 0x7f888c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.074+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f88a0098040 con 0x7f88b0070990 2024-07-26T06:18:33.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.074+0000 7f88ae000640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f888c0763f0 0x7f888c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.075+0000 7f88ae000640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f888c0763f0 0x7f888c0788b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f88980012e0 tx=0x7f8898007d30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:33.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.078+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88b00b5e40 con 0x7f88b0070990 2024-07-26T06:18:33.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.287+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f88b00b4a50 con 0x7f88b0070990 2024-07-26T06:18:33.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.288+0000 7f88a6a00640 1 -- 172.21.15.163:0/510691586 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f88a0014030 con 0x7f88b0070990 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f888c0763f0 msgr2=0x7f888c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f888c0763f0 0x7f888c0788b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f88980012e0 tx=0x7f8898007d30 comp rx=0 tx=0).stop 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 msgr2=0x7f88b00b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 0x7f88b00b0a30 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c950 tx=0x7f88a000ce10 comp rx=0 tx=0).stop 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 shutdown_connections 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f888c0763f0 0x7f888c0788b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b00ce040 0x7f88b014ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b0071990 0x7f88b00ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 --2- 172.21.15.163:0/510691586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f88b0070990 0x7f88b00b0a30 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 >> 172.21.15.163:0/510691586 conn(0x7f88b00b3060 msgr2=0x7f88b00b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:33.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.291+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 shutdown_connections 2024-07-26T06:18:33.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.292+0000 7f88afe00640 1 -- 172.21.15.163:0/510691586 wait complete. 2024-07-26T06:18:33.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create 'abcd@#' 2024-07-26T06:18:33.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 -- 172.21.15.163:0/3434158078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c4071c40 msgr2=0x7f21c40b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 --2- 172.21.15.163:0/3434158078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c4071c40 0x7f21c40b3710 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f21c406ce80 tx=0x7f21bc02aa30 comp rx=0 tx=0).stop 2024-07-26T06:18:33.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 -- 172.21.15.163:0/3434158078 shutdown_connections 2024-07-26T06:18:33.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 --2- 172.21.15.163:0/3434158078 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c4071c40 0x7f21c40b3710 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 --2- 172.21.15.163:0/3434158078 >> v2:172.21.15.164:3300/0 conn(0x7f21c41556f0 0x7f21c40cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 -- 172.21.15.163:0/3434158078 >> 172.21.15.163:0/3434158078 conn(0x7f21c40b0e30 msgr2=0x7f21c40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:33.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 -- 172.21.15.163:0/3434158078 shutdown_connections 2024-07-26T06:18:33.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.511+0000 7f21c3e00640 1 -- 172.21.15.163:0/3434158078 wait complete. 2024-07-26T06:18:33.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.513+0000 7f21c3e00640 1 Processor -- start 2024-07-26T06:18:33.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.513+0000 7f21c3e00640 1 -- start start 2024-07-26T06:18:33.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.514+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21c4071c40 0x7f21c41f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21c40ce2d0 0x7f21c414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c41556f0 0x7f21c4158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f21c40b3fe0 con 0x7f21c41556f0 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f21c40b3e60 con 0x7f21c4071c40 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f21c40b4160 con 0x7f21c40ce2d0 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c41556f0 0x7f21c4158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c41556f0 0x7f21c4158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55476/0 (socket says 172.21.15.163:55476) 2024-07-26T06:18:33.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 -- 172.21.15.163:0/3568658734 learned_addr learned my addr 172.21.15.163:0/3568658734 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:33.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c2a00640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21c4071c40 0x7f21c41f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21ba000640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21c40ce2d0 0x7f21c414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 -- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21c40ce2d0 msgr2=0x7f21c414ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21c40ce2d0 0x7f21c414ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 -- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21c4071c40 msgr2=0x7f21c41f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21c4071c40 0x7f21c41f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c3400640 1 -- 172.21.15.163:0/3568658734 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c4158f60 con 0x7f21c41556f0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21c2a00640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21c4071c40 0x7f21c41f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.515+0000 7f21ba000640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21c40ce2d0 0x7f21c414ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.516+0000 7f21c3400640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21c41556f0 0x7f21c4158660 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009d60 tx=0x7f21bc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.516+0000 7f21bb400640 1 -- 172.21.15.163:0/3568658734 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc047070 con 0x7f21c41556f0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.516+0000 7f21bb400640 1 -- 172.21.15.163:0/3568658734 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21bc004250 con 0x7f21c41556f0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.516+0000 7f21bb400640 1 -- 172.21.15.163:0/3568658734 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc033050 con 0x7f21c41556f0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.516+0000 7f21c3e00640 1 -- 172.21.15.163:0/3568658734 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c4159240 con 0x7f21c41556f0 2024-07-26T06:18:33.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.516+0000 7f21c3e00640 1 -- 172.21.15.163:0/3568658734 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c4159800 con 0x7f21c41556f0 2024-07-26T06:18:33.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.518+0000 7f21c3e00640 1 -- 172.21.15.163:0/3568658734 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c40b46f0 con 0x7f21c41556f0 2024-07-26T06:18:33.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.519+0000 7f21bb400640 1 -- 172.21.15.163:0/3568658734 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f21bc04a070 con 0x7f21c41556f0 2024-07-26T06:18:33.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.520+0000 7f21bb400640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21a00763f0 0x7f21a00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.520+0000 7f21bb400640 1 -- 172.21.15.163:0/3568658734 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f21bc0b8db0 con 0x7f21c41556f0 2024-07-26T06:18:33.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.520+0000 7f21c2a00640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21a00763f0 0x7f21a00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.521+0000 7f21c2a00640 1 --2- 172.21.15.163:0/3568658734 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21a00763f0 0x7f21a00788b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f21b40058c0 tx=0x7f21b4005c90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:33.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.524+0000 7f21bb400640 1 -- 172.21.15.163:0/3568658734 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f21bc0bd070 con 0x7f21c41556f0 2024-07-26T06:18:33.755 INFO:teuthology.orchestra.run.smithi163.stderr:Invalid command: invalid chars #@ in abcd@# 2024-07-26T06:18:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:fs volume create [] : Create a CephFS volume 2024-07-26T06:18:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:Error EINVAL: invalid command 2024-07-26T06:18:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-07-26T06:18:33.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create testvol 2024-07-26T06:18:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0499800640 1 -- 172.21.15.163:0/493974755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0478004d40 msgr2=0x7f04780051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0499800640 1 --2- 172.21.15.163:0/493974755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0478004d40 0x7f04780051a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f048c009ed0 tx=0x7f0478005830 comp rx=0 tx=0).stop 2024-07-26T06:18:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0498e00640 1 -- 172.21.15.163:0/493974755 reap_dead start 2024-07-26T06:18:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0499800640 1 -- 172.21.15.163:0/493974755 shutdown_connections 2024-07-26T06:18:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0499800640 1 -- 172.21.15.163:0/493974755 >> 172.21.15.163:0/493974755 conn(0x7f04940b0e30 msgr2=0x7f04940b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0499800640 1 -- 172.21.15.163:0/493974755 shutdown_connections 2024-07-26T06:18:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.984+0000 7f0499800640 1 -- 172.21.15.163:0/493974755 wait complete. 2024-07-26T06:18:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.986+0000 7f0499800640 1 Processor -- start 2024-07-26T06:18:33.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.986+0000 7f0499800640 1 -- start start 2024-07-26T06:18:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0499800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0478004d40 0x7f04941f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0499800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f049413d9f0 0x7f0494071860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0499800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 0x7f04941417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0499800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0494166310 con 0x7f0494143da0 2024-07-26T06:18:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0499800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0494166190 con 0x7f049413d9f0 2024-07-26T06:18:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0499800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0494166490 con 0x7f0478004d40 2024-07-26T06:18:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0498e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 0x7f04941417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0498e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 0x7f04941417f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55496/0 (socket says 172.21.15.163:55496) 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.987+0000 7f0498e00640 1 -- 172.21.15.163:0/803922685 learned_addr learned my addr 172.21.15.163:0/803922685 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0498e00640 1 -- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0478004d40 msgr2=0x7f04941f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0493e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0478004d40 0x7f04941f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f048b400640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f049413d9f0 0x7f0494071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0498e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0478004d40 0x7f04941f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0498e00640 1 -- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f049413d9f0 msgr2=0x7f0494071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0498e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f049413d9f0 0x7f0494071860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0498e00640 1 -- 172.21.15.163:0/803922685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04941f8200 con 0x7f0494143da0 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0493e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0478004d40 0x7f04941f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0498e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 0x7f04941417f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f048c009ed0 tx=0x7f048c034b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c043070 con 0x7f0494143da0 2024-07-26T06:18:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f048c034c80 con 0x7f0494143da0 2024-07-26T06:18:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c038040 con 0x7f0494143da0 2024-07-26T06:18:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04941f7960 con 0x7f0494143da0 2024-07-26T06:18:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.988+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04941f7f20 con 0x7f0494143da0 2024-07-26T06:18:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.990+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04940cbd40 con 0x7f0494143da0 2024-07-26T06:18:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.991+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f048c003bd0 con 0x7f0494143da0 2024-07-26T06:18:33.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.992+0000 7f0490c00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f04700763f0 0x7f04700788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:33.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.992+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f048c0b77e0 con 0x7f0494143da0 2024-07-26T06:18:33.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.992+0000 7f0493e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f04700763f0 0x7f04700788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:33.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.993+0000 7f0493e00640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f04700763f0 0x7f04700788b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f04840016e0 tx=0x7f0484006a10 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:33.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:33.997+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f048c0808b0 con 0x7f0494143da0 2024-07-26T06:18:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:34.212+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "testvol", "target": ["mon-mgr", ""]}) -- 0x7f04940ae8b0 con 0x7f04700763f0 2024-07-26T06:18:36.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.440+0000 7f0490c00640 1 -- 172.21.15.163:0/803922685 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 Volume created successfully (no MDS daemons created)) ==== 60+0+0 (secure 0 0 0) 0x7f04940ae8b0 con 0x7f04700763f0 2024-07-26T06:18:36.441 INFO:teuthology.orchestra.run.smithi163.stderr:Volume created successfully (no MDS daemons created) 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f04700763f0 msgr2=0x7f04700788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f04700763f0 0x7f04700788b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f04840016e0 tx=0x7f0484006a10 comp rx=0 tx=0).stop 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 msgr2=0x7f04941417f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 0x7f04941417f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f048c009ed0 tx=0x7f048c034b90 comp rx=0 tx=0).stop 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 shutdown_connections 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f04700763f0 0x7f04700788b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0494143da0 0x7f04941417f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f049413d9f0 0x7f0494071860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 --2- 172.21.15.163:0/803922685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0478004d40 0x7f04941f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.444+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 >> 172.21.15.163:0/803922685 conn(0x7f04940b0e30 msgr2=0x7f04940ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:36.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.445+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 shutdown_connections 2024-07-26T06:18:36.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.445+0000 7f0499800640 1 -- 172.21.15.163:0/803922685 wait complete. 2024-07-26T06:18:36.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create testvol 'abcd@#' 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.659+0000 7fae86600640 1 -- 172.21.15.163:0/55458935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 msgr2=0x7fae800b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.659+0000 7fae86600640 1 --2- 172.21.15.163:0/55458935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 0x7fae800b2130 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005fd0 tx=0x7fae6c02a5d0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.660+0000 7fae86600640 1 -- 172.21.15.163:0/55458935 shutdown_connections 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.660+0000 7fae86600640 1 --2- 172.21.15.163:0/55458935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 0x7fae800b2130 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.660+0000 7fae86600640 1 -- 172.21.15.163:0/55458935 >> 172.21.15.163:0/55458935 conn(0x7fae800b0e30 msgr2=0x7fae800b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.660+0000 7fae86600640 1 -- 172.21.15.163:0/55458935 shutdown_connections 2024-07-26T06:18:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.660+0000 7fae86600640 1 -- 172.21.15.163:0/55458935 wait complete. 2024-07-26T06:18:36.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.661+0000 7fae86600640 1 Processor -- start 2024-07-26T06:18:36.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.661+0000 7fae86600640 1 -- start start 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.662+0000 7fae86600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 0x7fae8014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.662+0000 7fae86600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae800cdc40 0x7fae800b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.662+0000 7fae86600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae80138060 0x7fae8014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.662+0000 7fae86600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fae800b2d80 con 0x7fae800b4420 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.662+0000 7fae86600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fae8006f500 con 0x7fae800cdc40 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.662+0000 7fae86600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fae800b2f00 con 0x7fae80138060 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 0x7fae8014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 0x7fae8014d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54662/0 (socket says 172.21.15.163:54662) 2024-07-26T06:18:36.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 -- 172.21.15.163:0/4113798009 learned_addr learned my addr 172.21.15.163:0/4113798009 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 -- 172.21.15.163:0/4113798009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae80138060 msgr2=0x7fae8014ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae80138060 0x7fae8014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae7fe00640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae800cdc40 0x7fae800b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 -- 172.21.15.163:0/4113798009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae800cdc40 msgr2=0x7fae800b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae800cdc40 0x7fae800b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 -- 172.21.15.163:0/4113798009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae80135660 con 0x7fae800b4420 2024-07-26T06:18:36.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.663+0000 7fae85200640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fae800b4420 0x7fae8014d2f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fae7400c430 tx=0x7fae7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:36.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.664+0000 7fae7cc00640 1 -- 172.21.15.163:0/4113798009 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74011790 con 0x7fae800b4420 2024-07-26T06:18:36.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.664+0000 7fae7cc00640 1 -- 172.21.15.163:0/4113798009 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae74017070 con 0x7fae800b4420 2024-07-26T06:18:36.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.664+0000 7fae7cc00640 1 -- 172.21.15.163:0/4113798009 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74011d00 con 0x7fae800b4420 2024-07-26T06:18:36.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.664+0000 7fae86600640 1 -- 172.21.15.163:0/4113798009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae801358d0 con 0x7fae800b4420 2024-07-26T06:18:36.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.664+0000 7fae86600640 1 -- 172.21.15.163:0/4113798009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae80135e90 con 0x7fae800b4420 2024-07-26T06:18:36.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.665+0000 7fae86600640 1 -- 172.21.15.163:0/4113798009 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae800cd530 con 0x7fae800b4420 2024-07-26T06:18:36.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.666+0000 7fae7cc00640 1 -- 172.21.15.163:0/4113798009 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fae7401c460 con 0x7fae800b4420 2024-07-26T06:18:36.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.667+0000 7fae7cc00640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fae600763f0 0x7fae600788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:36.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.667+0000 7fae7cc00640 1 -- 172.21.15.163:0/4113798009 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 6543+0+0 (secure 0 0 0) 0x7fae74097b60 con 0x7fae800b4420 2024-07-26T06:18:36.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.667+0000 7fae7fe00640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fae600763f0 0x7fae600788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:36.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.668+0000 7fae7fe00640 1 --2- 172.21.15.163:0/4113798009 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fae600763f0 0x7fae600788b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae6c033000 tx=0x7fae6c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:36.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:36.672+0000 7fae7cc00640 1 -- 172.21.15.163:0/4113798009 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fae74061070 con 0x7fae800b4420 2024-07-26T06:18:36.907 INFO:teuthology.orchestra.run.smithi163.stderr:Invalid command: invalid chars @# in abcd@# 2024-07-26T06:18:36.907 INFO:teuthology.orchestra.run.smithi163.stderr:fs subvolumegroup create [] [] [] [] [] : Create a CephFS subvolume group in a volume, and optionally, with a specific data pool layout, and a specific numeric mode 2024-07-26T06:18:36.907 INFO:teuthology.orchestra.run.smithi163.stderr:Error EINVAL: invalid command 2024-07-26T06:18:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-07-26T06:18:36.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create testvol 'abcd@#' 2024-07-26T06:18:37.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40be00640 1 -- 172.21.15.163:0/3813901309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3f0004d40 msgr2=0x7fa3f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40be00640 1 --2- 172.21.15.163:0/3813901309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa3f0004d40 0x7fa3f00051c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa404001dc0 tx=0x7fa3f0005830 comp rx=0 tx=0).stop 2024-07-26T06:18:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40b400640 1 -- 172.21.15.163:0/3813901309 reap_dead start 2024-07-26T06:18:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40be00640 1 -- 172.21.15.163:0/3813901309 shutdown_connections 2024-07-26T06:18:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40be00640 1 -- 172.21.15.163:0/3813901309 >> 172.21.15.163:0/3813901309 conn(0x7fa40c0b0e30 msgr2=0x7fa40c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40be00640 1 -- 172.21.15.163:0/3813901309 shutdown_connections 2024-07-26T06:18:37.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.132+0000 7fa40be00640 1 -- 172.21.15.163:0/3813901309 wait complete. 2024-07-26T06:18:37.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.134+0000 7fa40be00640 1 Processor -- start 2024-07-26T06:18:37.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.134+0000 7fa40be00640 1 -- start start 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.135+0000 7fa40be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3f0004d40 0x7fa40c0c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.135+0000 7fa40be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40c1f42c0 0x7fa40c1461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.135+0000 7fa40be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa40c0c5d30 0x7fa40c1f6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.135+0000 7fa40be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c0adc70 con 0x7fa40c0c5d30 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.135+0000 7fa40be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c0adaf0 con 0x7fa3f0004d40 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.135+0000 7fa40be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa40c0b3e60 con 0x7fa40c1f42c0 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa40c0c5d30 0x7fa40c1f6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa40c0c5d30 0x7fa40c1f6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54672/0 (socket says 172.21.15.163:54672) 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 -- 172.21.15.163:0/777735688 learned_addr learned my addr 172.21.15.163:0/777735688 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:37.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40aa00640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3f0004d40 0x7fa40c0c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa403e00640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40c1f42c0 0x7fa40c1461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 -- 172.21.15.163:0/777735688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40c1f42c0 msgr2=0x7fa40c1461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40c1f42c0 0x7fa40c1461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 -- 172.21.15.163:0/777735688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3f0004d40 msgr2=0x7fa40c0c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3f0004d40 0x7fa40c0c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 -- 172.21.15.163:0/777735688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40c0c5120 con 0x7fa40c0c5d30 2024-07-26T06:18:37.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.136+0000 7fa40b400640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa40c0c5d30 0x7fa40c1f6c50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa40402acc0 tx=0x7fa4040014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.137+0000 7fa402a00640 1 -- 172.21.15.163:0/777735688 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4040060b0 con 0x7fa40c0c5d30 2024-07-26T06:18:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.137+0000 7fa402a00640 1 -- 172.21.15.163:0/777735688 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa404006250 con 0x7fa40c0c5d30 2024-07-26T06:18:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.137+0000 7fa402a00640 1 -- 172.21.15.163:0/777735688 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404006680 con 0x7fa40c0c5d30 2024-07-26T06:18:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.137+0000 7fa40be00640 1 -- 172.21.15.163:0/777735688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40c143220 con 0x7fa40c0c5d30 2024-07-26T06:18:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.137+0000 7fa40be00640 1 -- 172.21.15.163:0/777735688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40c1437e0 con 0x7fa40c0c5d30 2024-07-26T06:18:37.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.138+0000 7fa40be00640 1 -- 172.21.15.163:0/777735688 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40c1439e0 con 0x7fa40c0c5d30 2024-07-26T06:18:37.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.139+0000 7fa402a00640 1 -- 172.21.15.163:0/777735688 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fa404033480 con 0x7fa40c0c5d30 2024-07-26T06:18:37.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.140+0000 7fa402a00640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa3e80763f0 0x7fa3e80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.140+0000 7fa402a00640 1 -- 172.21.15.163:0/777735688 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 6543+0+0 (secure 0 0 0) 0x7fa4040b7900 con 0x7fa40c0c5d30 2024-07-26T06:18:37.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.140+0000 7fa40aa00640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa3e80763f0 0x7fa3e80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:37.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.141+0000 7fa40aa00640 1 --2- 172.21.15.163:0/777735688 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa3e80763f0 0x7fa3e80788b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0057c0 tx=0x7fa3fc009630 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:37.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.145+0000 7fa402a00640 1 -- 172.21.15.163:0/777735688 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa40c1439e0 con 0x7fa40c0c5d30 2024-07-26T06:18:37.363 INFO:teuthology.orchestra.run.smithi163.stderr:Invalid command: invalid chars #@ in abcd@# 2024-07-26T06:18:37.363 INFO:teuthology.orchestra.run.smithi163.stderr:fs subvolume create [] [] [] [] [] [] [--namespace-isolated] : Create a CephFS subvolume in a volume, and optionally, with a specific size (in bytes), a specific data pool layout, a specific mode, in a specific subvolume group and in separate RADOS namespace 2024-07-26T06:18:37.363 INFO:teuthology.orchestra.run.smithi163.stderr:Error EINVAL: invalid command 2024-07-26T06:18:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-07-26T06:18:37.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm testvol --yes-i-really-mean-it 2024-07-26T06:18:37.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 -- 172.21.15.163:0/1334022546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc8071c40 msgr2=0x7fdbc80b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:37.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/1334022546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc8071c40 0x7fdbc80b3710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0005fd0 tx=0x7fdbc002aa80 comp rx=0 tx=0).stop 2024-07-26T06:18:37.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 -- 172.21.15.163:0/1334022546 shutdown_connections 2024-07-26T06:18:37.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/1334022546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc8071c40 0x7fdbc80b3710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:37.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/1334022546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc81556f0 0x7fdbc80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:37.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 -- 172.21.15.163:0/1334022546 >> 172.21.15.163:0/1334022546 conn(0x7fdbc80b0e30 msgr2=0x7fdbc80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:37.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 -- 172.21.15.163:0/1334022546 shutdown_connections 2024-07-26T06:18:37.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.585+0000 7fdbc7e00640 1 -- 172.21.15.163:0/1334022546 wait complete. 2024-07-26T06:18:37.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.587+0000 7fdbc7e00640 1 Processor -- start 2024-07-26T06:18:37.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.587+0000 7fdbc7e00640 1 -- start start 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 0x7fdbc81f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80ce2d0 0x7fdbc80aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc81556f0 0x7fdbc81464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc80b4020 con 0x7fdbc8071c40 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc80b3ea0 con 0x7fdbc81556f0 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdbc80b41a0 con 0x7fdbc80ce2d0 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.588+0000 7fdbc6a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 0x7fdbc81f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 0x7fdbc81f4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54686/0 (socket says 172.21.15.163:54686) 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 -- 172.21.15.163:0/2751710418 learned_addr learned my addr 172.21.15.163:0/2751710418 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:37.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 -- 172.21.15.163:0/2751710418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80ce2d0 msgr2=0x7fdbc80aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:37.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80ce2d0 0x7fdbc80aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:37.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 -- 172.21.15.163:0/2751710418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc81556f0 msgr2=0x7fdbc81464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:37.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc81556f0 0x7fdbc81464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:37.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 -- 172.21.15.163:0/2751710418 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc81f3800 con 0x7fdbc8071c40 2024-07-26T06:18:37.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbc6a00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 0x7fdbc81f4b80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:37.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800cd80 con 0x7fdbc8071c40 2024-07-26T06:18:37.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.589+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbb8004020 con 0x7fdbc8071c40 2024-07-26T06:18:37.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.590+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8004470 con 0x7fdbc8071c40 2024-07-26T06:18:37.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.590+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc81f3ae0 con 0x7fdbc8071c40 2024-07-26T06:18:37.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.590+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc81f40a0 con 0x7fdbc8071c40 2024-07-26T06:18:37.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.591+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc80b4730 con 0x7fdbc8071c40 2024-07-26T06:18:37.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.592+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fdbb80035f0 con 0x7fdbc8071c40 2024-07-26T06:18:37.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.593+0000 7fdbbea00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdba80763f0 0x7fdba80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:37.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.593+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6543+0+0 (secure 0 0 0) 0x7fdbb8098890 con 0x7fdbc8071c40 2024-07-26T06:18:37.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.593+0000 7fdbc6000640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdba80763f0 0x7fdba80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:37.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.594+0000 7fdbc6000640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdba80763f0 0x7fdba80788b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000f60 tx=0x7fdbb40079b0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:37.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.597+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdbb8061b60 con 0x7fdbc8071c40 2024-07-26T06:18:37.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:37.827+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "testvol", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) -- 0x7fdbc80b34f0 con 0x7fdba80763f0 2024-07-26T06:18:37.837 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:18:37.837+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:18:37.923 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:18:40.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.875+0000 7fdbbea00640 1 -- 172.21.15.163:0/2751710418 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+78 (secure 0 0 0) 0x7fdbc80b34f0 con 0x7fdba80763f0 2024-07-26T06:18:40.877 INFO:teuthology.orchestra.run.smithi163.stdout:metadata pool: cephfs.testvol.meta data pool: ['cephfs.testvol.data'] removed 2024-07-26T06:18:40.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdba80763f0 msgr2=0x7fdba80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdba80763f0 0x7fdba80788b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000f60 tx=0x7fdbb40079b0 comp rx=0 tx=0).stop 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 msgr2=0x7fdbc81f4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 0x7fdbc81f4b80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800eeb0 comp rx=0 tx=0).stop 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 shutdown_connections 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdba80763f0 0x7fdba80788b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc81556f0 0x7fdbc81464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80ce2d0 0x7fdbc80aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:40.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 --2- 172.21.15.163:0/2751710418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbc8071c40 0x7fdbc81f4b80 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:40.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 >> 172.21.15.163:0/2751710418 conn(0x7fdbc80b0e30 msgr2=0x7fdbc8151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:40.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 shutdown_connections 2024-07-26T06:18:40.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:40.880+0000 7fdbc7e00640 1 -- 172.21.15.163:0/2751710418 wait complete. 2024-07-26T06:18:40.908 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:18:40.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:41.015 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:41.150 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:18:41.150 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:41.151 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:41.151 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:41.207 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:41.207 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:41.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.522+0000 7fc066200640 1 -- 172.21.15.163:0/3814012586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 msgr2=0x7fc0600b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.522+0000 7fc066200640 1 --2- 172.21.15.163:0/3814012586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600b6a00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc0600690b0 tx=0x7fc05802a8f0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 -- 172.21.15.163:0/3814012586 shutdown_connections 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 --2- 172.21.15.163:0/3814012586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600b6a00 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 --2- 172.21.15.163:0/3814012586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 0x7fc0600b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 --2- 172.21.15.163:0/3814012586 >> v2:172.21.15.164:3300/0 conn(0x7fc06015d5a0 0x7fc0600ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 -- 172.21.15.163:0/3814012586 >> 172.21.15.163:0/3814012586 conn(0x7fc0600cfda0 msgr2=0x7fc0600d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 -- 172.21.15.163:0/3814012586 shutdown_connections 2024-07-26T06:18:41.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.523+0000 7fc066200640 1 -- 172.21.15.163:0/3814012586 wait complete. 2024-07-26T06:18:41.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.524+0000 7fc066200640 1 Processor -- start 2024-07-26T06:18:41.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.524+0000 7fc066200640 1 -- start start 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.525+0000 7fc066200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.525+0000 7fc066200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 0x7fc06014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.525+0000 7fc066200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc06015d5a0 0x7fc06015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.525+0000 7fc066200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0600b4da0 con 0x7fc0600b1e80 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.525+0000 7fc066200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc0600b2630 con 0x7fc06015d5a0 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.525+0000 7fc066200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc0600b4f20 con 0x7fc0600ce830 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54734/0 (socket says 172.21.15.163:54734) 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 -- 172.21.15.163:0/1741086808 learned_addr learned my addr 172.21.15.163:0/1741086808 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:41.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc065800640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc06015d5a0 0x7fc06015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc05fe00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 0x7fc06014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 -- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 msgr2=0x7fc06014e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 0x7fc06014e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 -- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc06015d5a0 msgr2=0x7fc06015ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc06015d5a0 0x7fc06015ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc064e00640 1 -- 172.21.15.163:0/1741086808 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06015f7f0 con 0x7fc0600b1e80 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc05fe00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 0x7fc06014e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:41.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.526+0000 7fc065800640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc06015d5a0 0x7fc06015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:41.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.527+0000 7fc064e00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600c6e10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc05400d770 tx=0x7fc05400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:41.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.527+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054017070 con 0x7fc0600b1e80 2024-07-26T06:18:41.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.527+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc054009cc0 con 0x7fc0600b1e80 2024-07-26T06:18:41.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.527+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06015fad0 con 0x7fc0600b1e80 2024-07-26T06:18:41.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.527+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0540106c0 con 0x7fc0600b1e80 2024-07-26T06:18:41.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.527+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0601526d0 con 0x7fc0600b1e80 2024-07-26T06:18:41.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.528+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc06015fff0 con 0x7fc0600b1e80 2024-07-26T06:18:41.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.529+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fc05401a070 con 0x7fc0600b1e80 2024-07-26T06:18:41.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.531+0000 7fc05cc00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc0440763f0 0x7fc0440788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:41.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.531+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7fc054097b40 con 0x7fc0600b1e80 2024-07-26T06:18:41.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.531+0000 7fc05fe00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc0440763f0 0x7fc0440788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:41.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.532+0000 7fc05fe00640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc0440763f0 0x7fc0440788b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc060160e50 tx=0x7fc050005e50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:41.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.535+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc06015fff0 con 0x7fc0600b1e80 2024-07-26T06:18:41.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.754+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc0600ce2f0 con 0x7fc0600b1e80 2024-07-26T06:18:41.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.754+0000 7fc05cc00640 1 -- 172.21.15.163:0/1741086808 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v88) ==== 80+0+124 (secure 0 0 0) 0x7fc054061180 con 0x7fc0600b1e80 2024-07-26T06:18:41.755 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163:6834/2494990100 2024-07-27T06:18:37.837948+0000 2024-07-26T06:18:41.755 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163:6832/2494990100 2024-07-27T06:18:37.837948+0000 2024-07-26T06:18:41.756 INFO:teuthology.orchestra.run.smithi163.stderr:listed 2 entries 2024-07-26T06:18:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc0440763f0 msgr2=0x7fc0440788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc0440763f0 0x7fc0440788b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc060160e50 tx=0x7fc050005e50 comp rx=0 tx=0).stop 2024-07-26T06:18:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 msgr2=0x7fc0600c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600c6e10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc05400d770 tx=0x7fc05400dc30 comp rx=0 tx=0).stop 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 shutdown_connections 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc0440763f0 0x7fc0440788b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc06015d5a0 0x7fc06015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0600ce830 0x7fc06014e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 --2- 172.21.15.163:0/1741086808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0600b1e80 0x7fc0600c6e10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 >> 172.21.15.163:0/1741086808 conn(0x7fc0600cfda0 msgr2=0x7fc0600b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 shutdown_connections 2024-07-26T06:18:41.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:41.757+0000 7fc066200640 1 -- 172.21.15.163:0/1741086808 wait complete. 2024-07-26T06:18:41.778 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:18:41.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:41.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-07-26T06:18:41.877 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:18:41.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:41.878 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:18:41.930 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:41.930 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:41.930 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:41.988 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:41.988 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:42.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.255+0000 7f14cb200640 1 -- 172.21.15.163:0/1872526554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14b0003a00 msgr2=0x7f14b0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.255+0000 7f14cb200640 1 --2- 172.21.15.163:0/1872526554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14b0003a00 0x7f14b0003e20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00bb20 tx=0x7f14b0005830 comp rx=0 tx=0).stop 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.255+0000 7f14ca800640 1 -- 172.21.15.163:0/1872526554 reap_dead start 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.256+0000 7f14cb200640 1 -- 172.21.15.163:0/1872526554 shutdown_connections 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.256+0000 7f14cb200640 1 -- 172.21.15.163:0/1872526554 >> 172.21.15.163:0/1872526554 conn(0x7f14cc0b0e30 msgr2=0x7f14cc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.256+0000 7f14cb200640 1 -- 172.21.15.163:0/1872526554 shutdown_connections 2024-07-26T06:18:42.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.256+0000 7f14cb200640 1 -- 172.21.15.163:0/1872526554 wait complete. 2024-07-26T06:18:42.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.257+0000 7f14cb200640 1 Processor -- start 2024-07-26T06:18:42.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 -- start start 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b0003a00 0x7f14cc146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 0x7f14cc151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14cc14d750 0x7f14cc151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc073380 con 0x7f14cc1f5be0 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc15a1d0 con 0x7f14b0003a00 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.258+0000 7f14cb200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f14cc15a350 con 0x7f14cc14d750 2024-07-26T06:18:42.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 0x7f14cc151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 0x7f14cc151900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54744/0 (socket says 172.21.15.163:54744) 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 -- 172.21.15.163:0/3002869255 learned_addr learned my addr 172.21.15.163:0/3002869255 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c9e00640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b0003a00 0x7f14cc146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14ca800640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14cc14d750 0x7f14cc151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 -- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14cc14d750 msgr2=0x7f14cc151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14cc14d750 0x7f14cc151e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 -- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b0003a00 msgr2=0x7f14cc146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b0003a00 0x7f14cc146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.259+0000 7f14c1400640 1 -- 172.21.15.163:0/3002869255 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc1472d0 con 0x7f14cc1f5be0 2024-07-26T06:18:42.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.260+0000 7f14c1400640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 0x7f14cc151900 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f14b400d860 tx=0x7f14b400dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:42.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.260+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4019070 con 0x7f14cc1f5be0 2024-07-26T06:18:42.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.260+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14b400de50 con 0x7f14cc1f5be0 2024-07-26T06:18:42.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.260+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b400b660 con 0x7f14cc1f5be0 2024-07-26T06:18:42.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.260+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1f60e0 con 0x7f14cc1f5be0 2024-07-26T06:18:42.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.260+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1529d0 con 0x7f14cc1f5be0 2024-07-26T06:18:42.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.261+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc06a1f0 con 0x7f14cc1f5be0 2024-07-26T06:18:42.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.262+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f14b401c070 con 0x7f14cc1f5be0 2024-07-26T06:18:42.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.264+0000 7f14c2a00640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f14a80763f0 0x7f14a80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.264+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7f14b4097210 con 0x7f14cc1f5be0 2024-07-26T06:18:42.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.264+0000 7f14c9e00640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f14a80763f0 0x7f14a80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.265+0000 7f14c9e00640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f14a80763f0 0x7f14a80788b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f14bc036000 tx=0x7f14bc0046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:42.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.268+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14cc06a1f0 con 0x7f14cc1f5be0 2024-07-26T06:18:42.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.493+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f14cc0b4f50 con 0x7f14cc1f5be0 2024-07-26T06:18:42.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.494+0000 7f14c2a00640 1 -- 172.21.15.163:0/3002869255 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v88) ==== 80+0+124 (secure 0 0 0) 0x7f14cc1472d0 con 0x7f14cc1f5be0 2024-07-26T06:18:42.495 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163:6834/2494990100 2024-07-27T06:18:37.837948+0000 2024-07-26T06:18:42.495 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163:6832/2494990100 2024-07-27T06:18:37.837948+0000 2024-07-26T06:18:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:listed 2 entries 2024-07-26T06:18:42.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f14a80763f0 msgr2=0x7f14a80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f14a80763f0 0x7f14a80788b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f14bc036000 tx=0x7f14bc0046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 msgr2=0x7f14cc151900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 0x7f14cc151900 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f14b400d860 tx=0x7f14b400dd20 comp rx=0 tx=0).stop 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 shutdown_connections 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f14a80763f0 0x7f14a80788b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.496+0000 7f14cb200640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14cc14d750 0x7f14cc151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.497+0000 7f14cb200640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f14cc1f5be0 0x7f14cc151900 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.497+0000 7f14cb200640 1 --2- 172.21.15.163:0/3002869255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b0003a00 0x7f14cc146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.497+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 >> 172.21.15.163:0/3002869255 conn(0x7f14cc0b0e30 msgr2=0x7f14cc0cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.497+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 shutdown_connections 2024-07-26T06:18:42.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.497+0000 7f14cb200640 1 -- 172.21.15.163:0/3002869255 wait complete. 2024-07-26T06:18:42.517 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:18:42.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:42.517 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-07-26T06:18:42.617 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:18:42.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:18:42.617 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:18:42.667 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:42.667 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:42.668 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:42.724 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:42.724 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:18:42.724 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:42.724 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:42.781 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:42.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:42.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 -- 172.21.15.163:0/2100716892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e5015d2e0 msgr2=0x7f0e50072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 --2- 172.21.15.163:0/2100716892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e5015d2e0 0x7f0e50072890 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0e50160d10 tx=0x7f0e4402a560 comp rx=0 tx=0).stop 2024-07-26T06:18:42.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 -- 172.21.15.163:0/2100716892 shutdown_connections 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 --2- 172.21.15.163:0/2100716892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500b4da0 0x7f0e500b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 --2- 172.21.15.163:0/2100716892 >> v2:172.21.15.164:3300/0 conn(0x7f0e50071970 0x7f0e500b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 --2- 172.21.15.163:0/2100716892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e5015d2e0 0x7f0e50072890 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 -- 172.21.15.163:0/2100716892 >> 172.21.15.163:0/2100716892 conn(0x7f0e500af540 msgr2=0x7f0e500af920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 -- 172.21.15.163:0/2100716892 shutdown_connections 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.987+0000 7f0e55e00640 1 -- 172.21.15.163:0/2100716892 wait complete. 2024-07-26T06:18:42.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.989+0000 7f0e55e00640 1 Processor -- start 2024-07-26T06:18:42.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.989+0000 7f0e55e00640 1 -- start start 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e55e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e50071970 0x7f0e501e76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e55e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 0x7f0e501e7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e55e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5015d2e0 0x7f0e501f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e55e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0e500b31a0 con 0x7f0e500b4da0 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e55e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0e500b3020 con 0x7f0e50071970 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e55e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0e500b3320 con 0x7f0e5015d2e0 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e4fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 0x7f0e501e7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e4fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 0x7f0e501e7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54766/0 (socket says 172.21.15.163:54766) 2024-07-26T06:18:42.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.990+0000 7f0e4fe00640 1 -- 172.21.15.163:0/1473450792 learned_addr learned my addr 172.21.15.163:0/1473450792 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:42.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e54a00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e50071970 0x7f0e501e76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e55400640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5015d2e0 0x7f0e501f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4fe00640 1 -- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5015d2e0 msgr2=0x7f0e501f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4fe00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5015d2e0 0x7f0e501f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4fe00640 1 -- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e50071970 msgr2=0x7f0e501e76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4fe00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e50071970 0x7f0e501e76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4fe00640 1 -- 172.21.15.163:0/1473450792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e501f4dd0 con 0x7f0e500b4da0 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e54a00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e50071970 0x7f0e501e76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e55400640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5015d2e0 0x7f0e501f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4fe00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 0x7f0e501e7c20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c890 tx=0x7f0e3c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c004220 con 0x7f0e500b4da0 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e3c0079b0 con 0x7f0e500b4da0 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c007de0 con 0x7f0e500b4da0 2024-07-26T06:18:42.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.991+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e501f50b0 con 0x7f0e500b4da0 2024-07-26T06:18:42.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.992+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e501f5670 con 0x7f0e500b4da0 2024-07-26T06:18:42.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.993+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e500b28d0 con 0x7f0e500b4da0 2024-07-26T06:18:42.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.994+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f0e3c005020 con 0x7f0e500b4da0 2024-07-26T06:18:42.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.995+0000 7f0e4cc00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e300763f0 0x7f0e300788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:42.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.995+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7f0e3c097f60 con 0x7f0e500b4da0 2024-07-26T06:18:42.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.995+0000 7f0e54a00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e300763f0 0x7f0e300788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:42.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:42.996+0000 7f0e54a00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e300763f0 0x7f0e300788b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e44033000 tx=0x7f0e44004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:43.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.000+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e3c014030 con 0x7f0e500b4da0 2024-07-26T06:18:43.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.249+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e5015e8c0 con 0x7f0e500b4da0 2024-07-26T06:18:43.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.250+0000 7f0e4cc00640 1 -- 172.21.15.163:0/1473450792 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5234 (secure 0 0 0) 0x7f0e3c0615a0 con 0x7f0e500b4da0 2024-07-26T06:18:43.251 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:43.252 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":61,"btime":"2024-07-26T06:18:38:843334+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6720,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3979720531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3979720531},{"type":"v1","addr":"172.21.15.163:6834","nonce":3979720531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:21.433788+0000","modified":"2024-07-26T06:18:24.503309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6516":{"gid":6516,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6834/3348438490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3348438490},{"type":"v1","addr":"172.21.15.164:6834","nonce":3348438490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6516,"qdb_cluster":[6516]},"id":6}]} 2024-07-26T06:18:43.252 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 61 2024-07-26T06:18:43.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e300763f0 msgr2=0x7f0e300788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e300763f0 0x7f0e300788b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e44033000 tx=0x7f0e44004730 comp rx=0 tx=0).stop 2024-07-26T06:18:43.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 msgr2=0x7f0e501e7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 0x7f0e501e7c20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c890 tx=0x7f0e3c00cd50 comp rx=0 tx=0).stop 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 shutdown_connections 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e300763f0 0x7f0e300788b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5015d2e0 0x7f0e501f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e500b4da0 0x7f0e501e7c20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 --2- 172.21.15.163:0/1473450792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e50071970 0x7f0e501e76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 >> 172.21.15.163:0/1473450792 conn(0x7f0e500af540 msgr2=0x7f0e50071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 shutdown_connections 2024-07-26T06:18:43.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.255+0000 7f0e55e00640 1 -- 172.21.15.163:0/1473450792 wait complete. 2024-07-26T06:18:43.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:43.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 -- 172.21.15.163:0/1526326203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f838015d540 msgr2=0x7f83800d2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 --2- 172.21.15.163:0/1526326203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f838015d540 0x7f83800d2950 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f83800d2580 tx=0x7f837402a590 comp rx=0 tx=0).stop 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 -- 172.21.15.163:0/1526326203 shutdown_connections 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 --2- 172.21.15.163:0/1526326203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83800d2e90 0x7f83800cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 --2- 172.21.15.163:0/1526326203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f838015d540 0x7f83800d2950 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 -- 172.21.15.163:0/1526326203 >> 172.21.15.163:0/1526326203 conn(0x7f83800b4fa0 msgr2=0x7f83800b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 -- 172.21.15.163:0/1526326203 shutdown_connections 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.479+0000 7f8388200640 1 -- 172.21.15.163:0/1526326203 wait complete. 2024-07-26T06:18:43.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.481+0000 7f8388200640 1 Processor -- start 2024-07-26T06:18:43.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.481+0000 7f8388200640 1 -- start start 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8388200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 0x7f83800cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8388200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83800d2e90 0x7f83800c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8388200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838015d540 0x7f83800ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8388200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83800b7920 con 0x7f83800d0370 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8388200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f83800b77a0 con 0x7f83800d2e90 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8388200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f83800b7aa0 con 0x7f838015d540 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8385800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 0x7f83800cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8385800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 0x7f83800cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54774/0 (socket says 172.21.15.163:54774) 2024-07-26T06:18:43.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.482+0000 7f8385800640 1 -- 172.21.15.163:0/2369478630 learned_addr learned my addr 172.21.15.163:0/2369478630 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8386200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838015d540 0x7f83800ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8384e00640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83800d2e90 0x7f83800c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8385800640 1 -- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838015d540 msgr2=0x7f83800ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8385800640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838015d540 0x7f83800ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8385800640 1 -- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83800d2e90 msgr2=0x7f83800c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8385800640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83800d2e90 0x7f83800c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8385800640 1 -- 172.21.15.163:0/2369478630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838015c260 con 0x7f83800d0370 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8384e00640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83800d2e90 0x7f83800c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:43.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f8385800640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 0x7f83800cb870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f837402ab80 tx=0x7f8374033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:43.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.483+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374031d60 con 0x7f83800d0370 2024-07-26T06:18:43.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.484+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8374033e10 con 0x7f83800d0370 2024-07-26T06:18:43.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.484+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374005260 con 0x7f83800d0370 2024-07-26T06:18:43.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.484+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83800cbf00 con 0x7f83800d0370 2024-07-26T06:18:43.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.484+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83800cc4c0 con 0x7f83800d0370 2024-07-26T06:18:43.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.485+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83800cfe30 con 0x7f83800d0370 2024-07-26T06:18:43.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.486+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f8374005400 con 0x7f83800d0370 2024-07-26T06:18:43.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.487+0000 7f837d600640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f83600763f0 0x7f83600788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.487+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7f83740b8ae0 con 0x7f83800d0370 2024-07-26T06:18:43.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.487+0000 7f8384e00640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f83600763f0 0x7f83600788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.488+0000 7f8384e00640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f83600763f0 0x7f83600788b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83800aa240 tx=0x7f836c005e70 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:43.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.491+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f837403f030 con 0x7f83800d0370 2024-07-26T06:18:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.737+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83800c7ea0 con 0x7f83800d0370 2024-07-26T06:18:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.739+0000 7f837d600640 1 -- 172.21.15.163:0/2369478630 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5234 (secure 0 0 0) 0x7f83740821c0 con 0x7f83800d0370 2024-07-26T06:18:43.740 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:43.740 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":61,"btime":"2024-07-26T06:18:38:843334+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6720,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3979720531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3979720531},{"type":"v1","addr":"172.21.15.163:6834","nonce":3979720531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:21.433788+0000","modified":"2024-07-26T06:18:24.503309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6516":{"gid":6516,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6834/3348438490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3348438490},{"type":"v1","addr":"172.21.15.164:6834","nonce":3348438490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6516,"qdb_cluster":[6516]},"id":6}]} 2024-07-26T06:18:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 61 2024-07-26T06:18:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f83600763f0 msgr2=0x7f83600788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f83600763f0 0x7f83600788b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83800aa240 tx=0x7f836c005e70 comp rx=0 tx=0).stop 2024-07-26T06:18:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 msgr2=0x7f83800cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 0x7f83800cb870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f837402ab80 tx=0x7f8374033c20 comp rx=0 tx=0).stop 2024-07-26T06:18:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 shutdown_connections 2024-07-26T06:18:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f83600763f0 0x7f83600788b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838015d540 0x7f83800ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83800d2e90 0x7f83800c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.741+0000 7f8388200640 1 --2- 172.21.15.163:0/2369478630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f83800d0370 0x7f83800cb870 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.742+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 >> 172.21.15.163:0/2369478630 conn(0x7f83800b4fa0 msgr2=0x7f83800b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.742+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 shutdown_connections 2024-07-26T06:18:43.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.742+0000 7f8388200640 1 -- 172.21.15.163:0/2369478630 wait complete. 2024-07-26T06:18:43.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:43.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb633200640 1 -- 172.21.15.163:0/531062201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb618003a80 msgr2=0x7fb618003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb633200640 1 --2- 172.21.15.163:0/531062201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb618003a80 0x7fb618003ea0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fb6240016c0 tx=0x7fb618005830 comp rx=0 tx=0).stop 2024-07-26T06:18:43.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb632800640 1 -- 172.21.15.163:0/531062201 reap_dead start 2024-07-26T06:18:43.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb633200640 1 -- 172.21.15.163:0/531062201 shutdown_connections 2024-07-26T06:18:43.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb633200640 1 -- 172.21.15.163:0/531062201 >> 172.21.15.163:0/531062201 conn(0x7fb6340b0e30 msgr2=0x7fb6340b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:43.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb633200640 1 -- 172.21.15.163:0/531062201 shutdown_connections 2024-07-26T06:18:43.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.964+0000 7fb633200640 1 -- 172.21.15.163:0/531062201 wait complete. 2024-07-26T06:18:43.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.966+0000 7fb633200640 1 Processor -- start 2024-07-26T06:18:43.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.966+0000 7fb633200640 1 -- start start 2024-07-26T06:18:43.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb633200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb618003a80 0x7fb63414add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb633200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb63414f480 0x7fb63414f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb633200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 0x7fb6340cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb633200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb6340718c0 con 0x7fb6340cd0f0 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb633200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb634071740 con 0x7fb618003a80 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb633200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6340b3e20 con 0x7fb63414f480 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb632800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 0x7fb6340cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb632800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 0x7fb6340cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54792/0 (socket says 172.21.15.163:54792) 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb632800640 1 -- 172.21.15.163:0/2583361084 learned_addr learned my addr 172.21.15.163:0/2583361084 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb631e00640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb618003a80 0x7fb63414add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.968+0000 7fb631400640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb63414f480 0x7fb63414f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb632800640 1 -- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb63414f480 msgr2=0x7fb63414f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb632800640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb63414f480 0x7fb63414f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb632800640 1 -- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb618003a80 msgr2=0x7fb63414add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb632800640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb618003a80 0x7fb63414add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:43.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb632800640 1 -- 172.21.15.163:0/2583361084 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6341f7470 con 0x7fb6340cd0f0 2024-07-26T06:18:43.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb632800640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 0x7fb6340cd5a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb62c001330 tx=0x7fb62c00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:43.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c016020 con 0x7fb6340cd0f0 2024-07-26T06:18:43.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb62c00b840 con 0x7fb6340cd0f0 2024-07-26T06:18:43.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6341f7750 con 0x7fb6340cd0f0 2024-07-26T06:18:43.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00bcb0 con 0x7fb6340cd0f0 2024-07-26T06:18:43.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.969+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6341f7d10 con 0x7fb6340cd0f0 2024-07-26T06:18:43.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.970+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6340b4380 con 0x7fb6340cd0f0 2024-07-26T06:18:43.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.972+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb62c019020 con 0x7fb6340cd0f0 2024-07-26T06:18:43.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.973+0000 7fb62a000640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb6140763f0 0x7fb6140788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:43.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.973+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7fb62c097680 con 0x7fb6340cd0f0 2024-07-26T06:18:43.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.973+0000 7fb631e00640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb6140763f0 0x7fb6140788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:43.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.974+0000 7fb631e00640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb6140763f0 0x7fb6140788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb624032000 tx=0x7fb624001190 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:43.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:43.977+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb62c020030 con 0x7fb6340cd0f0 2024-07-26T06:18:44.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.183+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6340b34f0 con 0x7fb6340cd0f0 2024-07-26T06:18:44.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.186+0000 7fb62a000640 1 -- 172.21.15.163:0/2583361084 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16815 (secure 0 0 0) 0x7fb62c060e60 con 0x7fb6340cd0f0 2024-07-26T06:18:44.187 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:44.188 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":88,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:40.851475+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:19.305919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:20.414373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:6834/2494990100":"2024-07-27T06:18:37.837948+0000","172.21.15.163:6832/2494990100":"2024-07-27T06:18:37.837948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb6140763f0 msgr2=0x7fb6140788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb6140763f0 0x7fb6140788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb624032000 tx=0x7fb624001190 comp rx=0 tx=0).stop 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 msgr2=0x7fb6340cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 0x7fb6340cd5a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb62c001330 tx=0x7fb62c00dd50 comp rx=0 tx=0).stop 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 shutdown_connections 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb6140763f0 0x7fb6140788b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6340cd0f0 0x7fb6340cd5a0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb63414f480 0x7fb63414f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 --2- 172.21.15.163:0/2583361084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb618003a80 0x7fb63414add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 >> 172.21.15.163:0/2583361084 conn(0x7fb6340b0e30 msgr2=0x7fb634153d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:44.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 shutdown_connections 2024-07-26T06:18:44.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.189+0000 7fb633200640 1 -- 172.21.15.163:0/2583361084 wait complete. 2024-07-26T06:18:44.208 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:18:44.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 -- 172.21.15.163:0/423544741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 msgr2=0x7f810c0ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 --2- 172.21.15.163:0/423544741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c0ce2f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f80fc005fd0 tx=0x7f80fc02a4a0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 -- 172.21.15.163:0/423544741 shutdown_connections 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 --2- 172.21.15.163:0/423544741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0b1e80 0x7f810c0b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 --2- 172.21.15.163:0/423544741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c0ce2f0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 -- 172.21.15.163:0/423544741 >> 172.21.15.163:0/423544741 conn(0x7f810c0cfda0 msgr2=0x7f810c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:44.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 -- 172.21.15.163:0/423544741 shutdown_connections 2024-07-26T06:18:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.407+0000 7f810be00640 1 -- 172.21.15.163:0/423544741 wait complete. 2024-07-26T06:18:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.409+0000 7f810be00640 1 Processor -- start 2024-07-26T06:18:44.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.409+0000 7f810be00640 1 -- start start 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810c0b1e80 0x7f810c15fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0ce830 0x7f810c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c1f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f810c0b5130 con 0x7f810c15d5a0 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f810c0b4fb0 con 0x7f810c0b1e80 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f810c0b52b0 con 0x7f810c0ce830 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c1f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c1f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:54810/0 (socket says 172.21.15.163:54810) 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 -- 172.21.15.163:0/3718822302 learned_addr learned my addr 172.21.15.163:0/3718822302 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810aa00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810c0b1e80 0x7f810c15fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810a000640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0ce830 0x7f810c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 -- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0ce830 msgr2=0x7f810c0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0ce830 0x7f810c0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 -- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810c0b1e80 msgr2=0x7f810c15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810c0b1e80 0x7f810c15fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810b400640 1 -- 172.21.15.163:0/3718822302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c0caad0 con 0x7f810c15d5a0 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810aa00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810c0b1e80 0x7f810c15fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.410+0000 7f810a000640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0ce830 0x7f810c0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.411+0000 7f810b400640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c1f7040 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f810400b970 tx=0x7f810400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.411+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104019070 con 0x7f810c15d5a0 2024-07-26T06:18:44.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.411+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f810400cc50 con 0x7f810c15d5a0 2024-07-26T06:18:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.411+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81040146a0 con 0x7f810c15d5a0 2024-07-26T06:18:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.411+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f810c0cadb0 con 0x7f810c15d5a0 2024-07-26T06:18:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.411+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f810c0cb370 con 0x7f810c15d5a0 2024-07-26T06:18:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.412+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810c0cf5a0 con 0x7f810c15d5a0 2024-07-26T06:18:44.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.413+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f810401c070 con 0x7f810c15d5a0 2024-07-26T06:18:44.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.414+0000 7f8102a00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80ec0763f0 0x7f80ec0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.414+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7f81040977b0 con 0x7f810c15d5a0 2024-07-26T06:18:44.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.415+0000 7f810aa00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80ec0763f0 0x7f80ec0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.415+0000 7f810aa00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80ec0763f0 0x7f80ec0788b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f80fc033000 tx=0x7f80fc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:44.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.418+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f810c0cf5a0 con 0x7f810c15d5a0 2024-07-26T06:18:44.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.668+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f810c0c7d70 con 0x7f810c15d5a0 2024-07-26T06:18:44.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.670+0000 7f8102a00640 1 -- 172.21.15.163:0/3718822302 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5234 (secure 0 0 0) 0x7f8104060f10 con 0x7f810c15d5a0 2024-07-26T06:18:44.671 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:44.671 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":61,"btime":"2024-07-26T06:18:38:843334+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6720,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3979720531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3979720531},{"type":"v1","addr":"172.21.15.163:6834","nonce":3979720531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:21.433788+0000","modified":"2024-07-26T06:18:24.503309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6516":{"gid":6516,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6834/3348438490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3348438490},{"type":"v1","addr":"172.21.15.164:6834","nonce":3348438490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6516,"qdb_cluster":[6516]},"id":6}]} 2024-07-26T06:18:44.671 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 61 2024-07-26T06:18:44.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.672+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80ec0763f0 msgr2=0x7f80ec0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.672+0000 7f810be00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80ec0763f0 0x7f80ec0788b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f80fc033000 tx=0x7f80fc004730 comp rx=0 tx=0).stop 2024-07-26T06:18:44.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.672+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 msgr2=0x7f810c1f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.672+0000 7f810be00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c1f7040 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f810400b970 tx=0x7f810400be30 comp rx=0 tx=0).stop 2024-07-26T06:18:44.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 shutdown_connections 2024-07-26T06:18:44.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80ec0763f0 0x7f80ec0788b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f810c15d5a0 0x7f810c1f7040 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810c0ce830 0x7f810c0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 --2- 172.21.15.163:0/3718822302 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810c0b1e80 0x7f810c15fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 >> 172.21.15.163:0/3718822302 conn(0x7f810c0cfda0 msgr2=0x7f810c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:44.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 shutdown_connections 2024-07-26T06:18:44.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.673+0000 7f810be00640 1 -- 172.21.15.163:0/3718822302 wait complete. 2024-07-26T06:18:44.693 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:44.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 -- 172.21.15.163:0/3615072629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e415d6a0 msgr2=0x7fb4e40ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 --2- 172.21.15.163:0/3615072629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e415d6a0 0x7fb4e40ce680 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005fd0 tx=0x7fb4d402a540 comp rx=0 tx=0).stop 2024-07-26T06:18:44.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 -- 172.21.15.163:0/3615072629 shutdown_connections 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 --2- 172.21.15.163:0/3615072629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4e40d0870 0x7fb4e41610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 --2- 172.21.15.163:0/3615072629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e415d6a0 0x7fb4e40ce680 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 -- 172.21.15.163:0/3615072629 >> 172.21.15.163:0/3615072629 conn(0x7fb4e40cfd80 msgr2=0x7fb4e40d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 -- 172.21.15.163:0/3615072629 shutdown_connections 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.897+0000 7fb4e9400640 1 -- 172.21.15.163:0/3615072629 wait complete. 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.899+0000 7fb4e9400640 1 Processor -- start 2024-07-26T06:18:44.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.899+0000 7fb4e9400640 1 -- start start 2024-07-26T06:18:44.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e9400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4e40d0870 0x7fb4e4152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e9400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4e415d6a0 0x7fb4e40a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 0x7fb4e415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e9400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e40b5910 con 0x7fb4e4161630 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e9400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e40b5790 con 0x7fb4e40d0870 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e9400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb4e40d1910 con 0x7fb4e415d6a0 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 0x7fb4e415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e3200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 0x7fb4e415ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38616/0 (socket says 172.21.15.163:38616) 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e3200640 1 -- 172.21.15.163:0/2180804680 learned_addr learned my addr 172.21.15.163:0/2180804680 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e2800640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4e40d0870 0x7fb4e4152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.900+0000 7fb4e1e00640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4e415d6a0 0x7fb4e40a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e3200640 1 -- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4e415d6a0 msgr2=0x7fb4e40a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e3200640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4e415d6a0 0x7fb4e40a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e3200640 1 -- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4e40d0870 msgr2=0x7fb4e4152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:44.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e3200640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4e40d0870 0x7fb4e4152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:44.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e3200640 1 -- 172.21.15.163:0/2180804680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e40cbe90 con 0x7fb4e4161630 2024-07-26T06:18:44.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e3200640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 0x7fb4e415ec30 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c6e0 tx=0x7fb4dc00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc003b70 con 0x7fb4e4161630 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4dc003d30 con 0x7fb4e4161630 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc005330 con 0x7fb4e4161630 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e40cc170 con 0x7fb4e4161630 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.901+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e40cc730 con 0x7fb4e4161630 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.902+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e40d1e80 con 0x7fb4e4161630 2024-07-26T06:18:44.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.904+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb4dc0054d0 con 0x7fb4e4161630 2024-07-26T06:18:44.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.904+0000 7fb4daa00640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb4c40763f0 0x7fb4c40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:44.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.905+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7fb4dc0976f0 con 0x7fb4e4161630 2024-07-26T06:18:44.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.905+0000 7fb4e2800640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb4c40763f0 0x7fb4c40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:44.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.905+0000 7fb4e2800640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb4c40763f0 0x7fb4c40788b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4033000 tx=0x7fb4d4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:44.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:44.909+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb4dc060f70 con 0x7fb4e4161630 2024-07-26T06:18:45.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.141+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4e40b9280 con 0x7fb4e4161630 2024-07-26T06:18:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.142+0000 7fb4daa00640 1 -- 172.21.15.163:0/2180804680 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5234 (secure 0 0 0) 0x7fb4dc060870 con 0x7fb4e4161630 2024-07-26T06:18:45.143 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:45.144 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":61,"btime":"2024-07-26T06:18:38:843334+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6720,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3979720531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3979720531},{"type":"v1","addr":"172.21.15.163:6834","nonce":3979720531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:21.433788+0000","modified":"2024-07-26T06:18:24.503309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6516":{"gid":6516,"name":"b","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6834/3348438490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3348438490},{"type":"v1","addr":"172.21.15.164:6834","nonce":3348438490}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6516,"qdb_cluster":[6516]},"id":6}]} 2024-07-26T06:18:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 61 2024-07-26T06:18:45.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb4c40763f0 msgr2=0x7fb4c40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb4c40763f0 0x7fb4c40788b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4033000 tx=0x7fb4d4004730 comp rx=0 tx=0).stop 2024-07-26T06:18:45.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 msgr2=0x7fb4e415ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 0x7fb4e415ec30 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c6e0 tx=0x7fb4dc00cba0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 shutdown_connections 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb4c40763f0 0x7fb4c40788b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4e4161630 0x7fb4e415ec30 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4e415d6a0 0x7fb4e40a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 --2- 172.21.15.163:0/2180804680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4e40d0870 0x7fb4e4152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.145+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 >> 172.21.15.163:0/2180804680 conn(0x7fb4e40cfd80 msgr2=0x7fb4e4160800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.146+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 shutdown_connections 2024-07-26T06:18:45.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.146+0000 7fb4e9400640 1 -- 172.21.15.163:0/2180804680 wait complete. 2024-07-26T06:18:45.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:45.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2227685400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 msgr2=0x7ff5bc0b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2227685400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 0x7ff5bc0b2130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800a940 tx=0x7ff5a802b850 comp rx=0 tx=0).stop 2024-07-26T06:18:45.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2227685400 shutdown_connections 2024-07-26T06:18:45.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2227685400 >> v2:172.21.15.164:3300/0 conn(0x7ff5bc0cdc40 0x7ff5bc0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2227685400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 0x7ff5bc0b2130 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2227685400 >> 172.21.15.163:0/2227685400 conn(0x7ff5bc0b0e30 msgr2=0x7ff5bc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:45.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2227685400 shutdown_connections 2024-07-26T06:18:45.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.359+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2227685400 wait complete. 2024-07-26T06:18:45.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.361+0000 7ff5bbe00640 1 Processor -- start 2024-07-26T06:18:45.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.362+0000 7ff5bbe00640 1 -- start start 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bbe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 0x7ff5bc1f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 0x7ff5bc153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bbe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5bc074090 0x7ff5bc0c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bbe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc0b2bb0 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bbe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc06f500 con 0x7ff5bc074090 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bbe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff5bc0b2d30 con 0x7ff5bc0b4420 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5ba000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 0x7ff5bc153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5ba000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 0x7ff5bc153d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38632/0 (socket says 172.21.15.163:38632) 2024-07-26T06:18:45.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5ba000640 1 -- 172.21.15.163:0/2323774111 learned_addr learned my addr 172.21.15.163:0/2323774111 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:45.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5baa00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 0x7ff5bc1f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:45.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.363+0000 7ff5bb400640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5bc074090 0x7ff5bc0c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5ba000640 1 -- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 msgr2=0x7ff5bc1f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5ba000640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 0x7ff5bc1f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5ba000640 1 -- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5bc074090 msgr2=0x7ff5bc0c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5ba000640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5bc074090 0x7ff5bc0c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5ba000640 1 -- 172.21.15.163:0/2323774111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5bc154260 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5ba000640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 0x7ff5bc153d20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8000c00 tx=0x7ff5a8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8031750 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5a80319e0 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8031e30 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.364+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5bc0c4140 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.365+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5bc0c4700 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.366+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5bc0cd530 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.367+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7ff5a8033440 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.368+0000 7ff5b2a00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff59c0763f0 0x7ff59c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.368+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7ff5a80b91c0 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.368+0000 7ff5baa00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff59c0763f0 0x7ff59c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:45.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.369+0000 7ff5baa00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff59c0763f0 0x7ff59c0788b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0c7e40 tx=0x7ff5ac009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:45.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.372+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff5a8041030 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.583+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5bc0b14c0 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.585+0000 7ff5b2a00640 1 -- 172.21.15.163:0/2323774111 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16815 (secure 0 0 0) 0x7ff5a8082910 con 0x7ff5bc0cdc40 2024-07-26T06:18:45.586 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:45.586 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":88,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:40.851475+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:19.305919+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:20.414373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:6834/2494990100":"2024-07-27T06:18:37.837948+0000","172.21.15.163:6832/2494990100":"2024-07-27T06:18:37.837948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.587+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff59c0763f0 msgr2=0x7ff59c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.587+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff59c0763f0 0x7ff59c0788b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0c7e40 tx=0x7ff5ac009040 comp rx=0 tx=0).stop 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 msgr2=0x7ff5bc153d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 0x7ff5bc153d20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8000c00 tx=0x7ff5a8003b90 comp rx=0 tx=0).stop 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 shutdown_connections 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff59c0763f0 0x7ff59c0788b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5bc074090 0x7ff5bc0c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5bc0cdc40 0x7ff5bc153d20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 --2- 172.21.15.163:0/2323774111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5bc0b4420 0x7ff5bc1f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 >> 172.21.15.163:0/2323774111 conn(0x7ff5bc0b0e30 msgr2=0x7ff5bc0b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:45.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 shutdown_connections 2024-07-26T06:18:45.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.588+0000 7ff5bbe00640 1 -- 172.21.15.163:0/2323774111 wait complete. 2024-07-26T06:18:45.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:18:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 -- 172.21.15.163:0/1455101450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a40ce2d0 msgr2=0x7fa1a4071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/1455101450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a40ce2d0 0x7fa1a4071530 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa18c005fd0 tx=0x7fa18c02a530 comp rx=0 tx=0).stop 2024-07-26T06:18:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 -- 172.21.15.163:0/1455101450 shutdown_connections 2024-07-26T06:18:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/1455101450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a40ce2d0 0x7fa1a4071530 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/1455101450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1a41556f0 0x7fa1a40cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 -- 172.21.15.163:0/1455101450 >> 172.21.15.163:0/1455101450 conn(0x7fa1a40b0e30 msgr2=0x7fa1a40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:45.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 -- 172.21.15.163:0/1455101450 shutdown_connections 2024-07-26T06:18:45.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.811+0000 7fa1a9c00640 1 -- 172.21.15.163:0/1455101450 wait complete. 2024-07-26T06:18:45.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.813+0000 7fa1a9c00640 1 Processor -- start 2024-07-26T06:18:45.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.813+0000 7fa1a9c00640 1 -- start start 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a9c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 0x7fa1a40cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a9c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1a40ce2d0 0x7fa1a4150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a9c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1a41556f0 0x7fa1a40a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a9c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a415f310 con 0x7fa1a4071a70 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a9c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a4071ee0 con 0x7fa1a40ce2d0 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a9c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a415f490 con 0x7fa1a41556f0 2024-07-26T06:18:45.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 0x7fa1a40cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:45.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 0x7fa1a40cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38654/0 (socket says 172.21.15.163:38654) 2024-07-26T06:18:45.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.814+0000 7fa1a3e00640 1 -- 172.21.15.163:0/3001445150 learned_addr learned my addr 172.21.15.163:0/3001445150 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a3e00640 1 -- 172.21.15.163:0/3001445150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1a41556f0 msgr2=0x7fa1a40a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a3e00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1a41556f0 0x7fa1a40a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a3e00640 1 -- 172.21.15.163:0/3001445150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1a40ce2d0 msgr2=0x7fa1a4150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a3e00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1a40ce2d0 0x7fa1a4150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a3e00640 1 -- 172.21.15.163:0/3001445150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a41f4260 con 0x7fa1a4071a70 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a3e00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 0x7fa1a40cbb80 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa19400c970 tx=0x7fa19400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194007bb0 con 0x7fa1a4071a70 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa194007e40 con 0x7fa1a4071a70 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a41f4540 con 0x7fa1a4071a70 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194005280 con 0x7fa1a4071a70 2024-07-26T06:18:45.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.815+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a41f4b00 con 0x7fa1a4071a70 2024-07-26T06:18:45.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.817+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1a4071530 con 0x7fa1a4071a70 2024-07-26T06:18:45.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.818+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fa194005420 con 0x7fa1a4071a70 2024-07-26T06:18:45.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.819+0000 7fa1a0c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa1800763f0 0x7fa1800788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:45.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.819+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5795+0+0 (secure 0 0 0) 0x7fa194097f70 con 0x7fa1a4071a70 2024-07-26T06:18:45.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.820+0000 7fa19b400640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa1800763f0 0x7fa1800788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:45.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.820+0000 7fa19b400640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa1800763f0 0x7fa1800788b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa18c033000 tx=0x7fa18c0046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:45.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:45.824+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa194014030 con 0x7fa1a4071a70 2024-07-26T06:18:46.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.082+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa1a40ced60 con 0x7fa1a4071a70 2024-07-26T06:18:46.084 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:18:46.084+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:18:46.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.088+0000 7fa1a0c00640 1 -- 172.21.15.163:0/3001445150 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) ==== 161+0+0 (secure 0 0 0) 0x7fa1940615b0 con 0x7fa1a4071a70 2024-07-26T06:18:46.089 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:18:46.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa1800763f0 msgr2=0x7fa1800788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:46.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa1800763f0 0x7fa1800788b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa18c033000 tx=0x7fa18c0046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 msgr2=0x7fa1a40cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 0x7fa1a40cbb80 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa19400c970 tx=0x7fa19400ce30 comp rx=0 tx=0).stop 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 shutdown_connections 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa1800763f0 0x7fa1800788b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1a41556f0 0x7fa1a40a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1a40ce2d0 0x7fa1a4150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 --2- 172.21.15.163:0/3001445150 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1a4071a70 0x7fa1a40cbb80 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.091+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 >> 172.21.15.163:0/3001445150 conn(0x7fa1a40b0e30 msgr2=0x7fa1a415d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.092+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 shutdown_connections 2024-07-26T06:18:46.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.092+0000 7fa1a9c00640 1 -- 172.21.15.163:0/3001445150 wait complete. 2024-07-26T06:18:46.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:18:46.114 INFO:tasks.ceph.mds.b.smithi164.stderr: -2784> 2024-07-26T06:18:23.923+0000 7f5535c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:18:46.203 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:18:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e02400640 1 -- 172.21.15.163:0/2816342997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6de40040d0 msgr2=0x7f6de4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e02400640 1 --2- 172.21.15.163:0/2816342997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6de40040d0 0x7f6de4004800 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6de800bb20 tx=0x7f6de4005830 comp rx=0 tx=0).stop 2024-07-26T06:18:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e01a00640 1 -- 172.21.15.163:0/2816342997 reap_dead start 2024-07-26T06:18:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e02400640 1 -- 172.21.15.163:0/2816342997 shutdown_connections 2024-07-26T06:18:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e02400640 1 -- 172.21.15.163:0/2816342997 >> 172.21.15.163:0/2816342997 conn(0x7f6dfc06f470 msgr2=0x7f6dfc06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e02400640 1 -- 172.21.15.163:0/2816342997 shutdown_connections 2024-07-26T06:18:46.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.321+0000 7f6e02400640 1 -- 172.21.15.163:0/2816342997 wait complete. 2024-07-26T06:18:46.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.323+0000 7f6e02400640 1 Processor -- start 2024-07-26T06:18:46.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.323+0000 7f6e02400640 1 -- start start 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e02400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de40040d0 0x7f6dfc151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e02400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dfc1569e0 0x7f6dfc0ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e02400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 0x7f6dfc06e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e02400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc15c600 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e02400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc0ce320 con 0x7f6de40040d0 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e02400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6dfc15c780 con 0x7f6dfc1569e0 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 0x7f6dfc06e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 0x7f6dfc06e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38668/0 (socket says 172.21.15.163:38668) 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 -- 172.21.15.163:0/2370515456 learned_addr learned my addr 172.21.15.163:0/2370515456 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:46.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6dfbe00640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dfc1569e0 0x7f6dfc0ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01000640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de40040d0 0x7f6dfc151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 -- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dfc1569e0 msgr2=0x7f6dfc0ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dfc1569e0 0x7f6dfc0ab5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 -- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de40040d0 msgr2=0x7f6dfc151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de40040d0 0x7f6dfc151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.324+0000 7f6e01a00640 1 -- 172.21.15.163:0/2370515456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dfc1f8200 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.325+0000 7f6e01a00640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 0x7f6dfc06e0a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0732f0 tx=0x7f6df400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.325+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4016070 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.325+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6df400ccc0 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.325+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dfc1f72a0 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.325+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40116d0 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.325+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dfc1f7860 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.326+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dfc1f3ce0 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.327+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f6df4019070 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.328+0000 7f6df8c00640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6ddc076420 0x7f6ddc0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:46.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.329+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5881+0+0 (secure 0 0 0) 0x7f6df4097e80 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.329+0000 7f6e01000640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6ddc076420 0x7f6ddc0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:46.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.329+0000 7f6e01000640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6ddc076420 0x7f6ddc0788e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6df0001330 tx=0x7f6df0001260 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:46.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.333+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6dfc1f3ce0 con 0x7f6dfc0abaf0 2024-07-26T06:18:46.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:46.583+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6dfc0d0b00 con 0x7f6dfc0abaf0 2024-07-26T06:18:47.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.101+0000 7f6df8c00640 1 -- 172.21.15.163:0/2370515456 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v63) ==== 104+0+0 (secure 0 0 0) 0x7f6df4061500 con 0x7f6dfc0abaf0 2024-07-26T06:18:47.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6ddc076420 msgr2=0x7f6ddc0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:47.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6ddc076420 0x7f6ddc0788e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6df0001330 tx=0x7f6df0001260 comp rx=0 tx=0).stop 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 msgr2=0x7f6dfc06e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 0x7f6dfc06e0a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0732f0 tx=0x7f6df400ed20 comp rx=0 tx=0).stop 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 shutdown_connections 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6ddc076420 0x7f6ddc0788e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6dfc0abaf0 0x7f6dfc06e0a0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dfc1569e0 0x7f6dfc0ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 --2- 172.21.15.163:0/2370515456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de40040d0 0x7f6dfc151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.105+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 >> 172.21.15.163:0/2370515456 conn(0x7f6dfc06f470 msgr2=0x7f6dfc1598a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.106+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 shutdown_connections 2024-07-26T06:18:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.106+0000 7f6e02400640 1 -- 172.21.15.163:0/2370515456 wait complete. 2024-07-26T06:18:47.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:18:47.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e5800640 1 -- 172.21.15.163:0/619490029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4004800 msgr2=0x7f43c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:47.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e5800640 1 --2- 172.21.15.163:0/619490029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43c4004800 0x7f43c4004c80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f43c800baf0 tx=0x7f43c4005830 comp rx=0 tx=0).stop 2024-07-26T06:18:47.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e4e00640 1 -- 172.21.15.163:0/619490029 reap_dead start 2024-07-26T06:18:47.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e5800640 1 -- 172.21.15.163:0/619490029 shutdown_connections 2024-07-26T06:18:47.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e5800640 1 -- 172.21.15.163:0/619490029 >> 172.21.15.163:0/619490029 conn(0x7f43e00cfda0 msgr2=0x7f43e00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:47.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e5800640 1 -- 172.21.15.163:0/619490029 shutdown_connections 2024-07-26T06:18:47.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.342+0000 7f43e5800640 1 -- 172.21.15.163:0/619490029 wait complete. 2024-07-26T06:18:47.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.344+0000 7f43e5800640 1 Processor -- start 2024-07-26T06:18:47.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.344+0000 7f43e5800640 1 -- start start 2024-07-26T06:18:47.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e5800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43c4004800 0x7f43e0151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:47.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e5800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e015edf0 0x7f43e015f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e5800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 0x7f43e00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e5800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43e00ce470 con 0x7f43e00ca5a0 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e5800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43e00ce2f0 con 0x7f43e015edf0 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e5800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f43e00ce5f0 con 0x7f43c4004800 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 0x7f43e00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 0x7f43e00caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38692/0 (socket says 172.21.15.163:38692) 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.345+0000 7f43e4e00640 1 -- 172.21.15.163:0/437120636 learned_addr learned my addr 172.21.15.163:0/437120636 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43dfe00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43c4004800 0x7f43e0151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43d7400640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e015edf0 0x7f43e015f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43e4e00640 1 -- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43c4004800 msgr2=0x7f43e0151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:47.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43e4e00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43c4004800 0x7f43e0151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:47.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43e4e00640 1 -- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e015edf0 msgr2=0x7f43e015f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:47.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43e4e00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e015edf0 0x7f43e015f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:47.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43e4e00640 1 -- 172.21.15.163:0/437120636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e0151660 con 0x7f43e00ca5a0 2024-07-26T06:18:47.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43dfe00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43c4004800 0x7f43e0151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43e4e00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 0x7f43e00caa20 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f43d8001330 tx=0x7f43d800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8017070 con 0x7f43e00ca5a0 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43d800ada0 con 0x7f43e00ca5a0 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.346+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d80072d0 con 0x7f43e00ca5a0 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.347+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e015f940 con 0x7f43e00ca5a0 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.347+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e015ff00 con 0x7f43e00ca5a0 2024-07-26T06:18:47.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.348+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e00b5120 con 0x7f43e00ca5a0 2024-07-26T06:18:47.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.349+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f43d8007680 con 0x7f43e00ca5a0 2024-07-26T06:18:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.350+0000 7f43dcc00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f43bc076420 0x7f43bc0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.350+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5881+0+0 (secure 0 0 0) 0x7f43d80978d0 con 0x7f43e00ca5a0 2024-07-26T06:18:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.350+0000 7f43dfe00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f43bc076420 0x7f43bc0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:47.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.351+0000 7f43dfe00640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f43bc076420 0x7f43bc0788e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43d0005b60 tx=0x7f43d0005c90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:47.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.354+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43d8061060 con 0x7f43e00ca5a0 2024-07-26T06:18:47.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:47.562+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f43e00d1ba0 con 0x7f43e00ca5a0 2024-07-26T06:18:48.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.105+0000 7f43dcc00640 1 -- 172.21.15.163:0/437120636 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) ==== 181+0+0 (secure 0 0 0) 0x7f43d8060960 con 0x7f43e00ca5a0 2024-07-26T06:18:48.106 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:18:48.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f43bc076420 msgr2=0x7f43bc0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:48.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f43bc076420 0x7f43bc0788e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43d0005b60 tx=0x7f43d0005c90 comp rx=0 tx=0).stop 2024-07-26T06:18:48.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 msgr2=0x7f43e00caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:48.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 0x7f43e00caa20 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f43d8001330 tx=0x7f43d800a430 comp rx=0 tx=0).stop 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 shutdown_connections 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f43bc076420 0x7f43bc0788e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.109+0000 7f43e5800640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43e00ca5a0 0x7f43e00caa20 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.110+0000 7f43e5800640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43e015edf0 0x7f43e015f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.110+0000 7f43e5800640 1 --2- 172.21.15.163:0/437120636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43c4004800 0x7f43e0151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.110+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 >> 172.21.15.163:0/437120636 conn(0x7f43e00cfda0 msgr2=0x7f43e015aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.110+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 shutdown_connections 2024-07-26T06:18:48.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.110+0000 7f43e5800640 1 -- 172.21.15.163:0/437120636 wait complete. 2024-07-26T06:18:48.139 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:18:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.337+0000 7f2a56200640 1 -- 172.21.15.163:0/834765588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500d0e90 msgr2=0x7f2a500b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.337+0000 7f2a56200640 1 --2- 172.21.15.163:0/834765588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500d0e90 0x7f2a500b6110 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c009830 tx=0x7f2a3c02a490 comp rx=0 tx=0).stop 2024-07-26T06:18:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.337+0000 7f2a56200640 1 -- 172.21.15.163:0/834765588 shutdown_connections 2024-07-26T06:18:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.337+0000 7f2a56200640 1 --2- 172.21.15.163:0/834765588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500d0e90 0x7f2a500b6110 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.337+0000 7f2a56200640 1 --2- 172.21.15.163:0/834765588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a500cd910 0x7f2a500cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.337+0000 7f2a56200640 1 -- 172.21.15.163:0/834765588 >> 172.21.15.163:0/834765588 conn(0x7f2a500cfd40 msgr2=0x7f2a500d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:48.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.338+0000 7f2a56200640 1 -- 172.21.15.163:0/834765588 shutdown_connections 2024-07-26T06:18:48.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.338+0000 7f2a56200640 1 -- 172.21.15.163:0/834765588 wait complete. 2024-07-26T06:18:48.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.339+0000 7f2a56200640 1 Processor -- start 2024-07-26T06:18:48.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.339+0000 7f2a56200640 1 -- start start 2024-07-26T06:18:48.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.340+0000 7f2a56200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 0x7f2a5014c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:48.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.340+0000 7f2a56200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a500cd910 0x7f2a500ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:48.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.340+0000 7f2a56200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a500d0e90 0x7f2a500ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.340+0000 7f2a56200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2a500d2d30 con 0x7f2a500b6650 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.340+0000 7f2a56200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2a500d2bb0 con 0x7f2a500d0e90 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.340+0000 7f2a56200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2a500b4d40 con 0x7f2a500cd910 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 0x7f2a5014c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 0x7f2a5014c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38704/0 (socket says 172.21.15.163:38704) 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 -- 172.21.15.163:0/3751146515 learned_addr learned my addr 172.21.15.163:0/3751146515 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 -- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a500cd910 msgr2=0x7f2a500ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a500cd910 0x7f2a500ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 -- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a500d0e90 msgr2=0x7f2a500ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a55800640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a500d0e90 0x7f2a500ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a500d0e90 0x7f2a500ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 -- 172.21.15.163:0/3751146515 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5014c550 con 0x7f2a500b6650 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.341+0000 7f2a54e00640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 0x7f2a5014c010 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2a4000d770 tx=0x7f2a4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.342+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a40017070 con 0x7f2a500b6650 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.342+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a40009d20 con 0x7f2a500b6650 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.342+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a400106c0 con 0x7f2a500b6650 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.342+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5015df10 con 0x7f2a500b6650 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.342+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5014c8e0 con 0x7f2a500b6650 2024-07-26T06:18:48.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.343+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a500b2910 con 0x7f2a500b6650 2024-07-26T06:18:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.344+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f2a4001a070 con 0x7f2a500b6650 2024-07-26T06:18:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.345+0000 7f2a4cc00640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2a300763f0 0x7f2a300788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.345+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5443+0+0 (secure 0 0 0) 0x7f2a40097a40 con 0x7f2a500b6650 2024-07-26T06:18:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.346+0000 7f2a4fe00640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2a300763f0 0x7f2a300788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.346+0000 7f2a4fe00640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2a300763f0 0x7f2a300788b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c034000 tx=0x7f2a3c0046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:48.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.350+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2a500b2910 con 0x7f2a500b6650 2024-07-26T06:18:48.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:48.559+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2a500cdcf0 con 0x7f2a500b6650 2024-07-26T06:18:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.120+0000 7f2a4cc00640 1 -- 172.21.15.163:0/3751146515 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) ==== 169+0+0 (secure 0 0 0) 0x7f2a400611e0 con 0x7f2a500b6650 2024-07-26T06:18:49.122 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:18:49.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2a300763f0 msgr2=0x7f2a300788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2a300763f0 0x7f2a300788b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c034000 tx=0x7f2a3c0046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 msgr2=0x7f2a5014c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 0x7f2a5014c010 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2a4000d770 tx=0x7f2a4000dc30 comp rx=0 tx=0).stop 2024-07-26T06:18:49.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 shutdown_connections 2024-07-26T06:18:49.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2a300763f0 0x7f2a300788b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a500d0e90 0x7f2a500ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a500cd910 0x7f2a500ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 --2- 172.21.15.163:0/3751146515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a500b6650 0x7f2a5014c010 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 >> 172.21.15.163:0/3751146515 conn(0x7f2a500cfd40 msgr2=0x7f2a5006d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:49.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.125+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 shutdown_connections 2024-07-26T06:18:49.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.126+0000 7f2a56200640 1 -- 172.21.15.163:0/3751146515 wait complete. 2024-07-26T06:18:49.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 -- 172.21.15.163:0/2359394971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 msgr2=0x7f06500b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 --2- 172.21.15.163:0/2359394971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b6bd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0648005fd0 tx=0x7f064802a780 comp rx=0 tx=0).stop 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 -- 172.21.15.163:0/2359394971 shutdown_connections 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 --2- 172.21.15.163:0/2359394971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b6bd0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 -- 172.21.15.163:0/2359394971 >> 172.21.15.163:0/2359394971 conn(0x7f06500cfda0 msgr2=0x7f06500d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 -- 172.21.15.163:0/2359394971 shutdown_connections 2024-07-26T06:18:49.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.342+0000 7f0656600640 1 -- 172.21.15.163:0/2359394971 wait complete. 2024-07-26T06:18:49.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.344+0000 7f0656600640 1 Processor -- start 2024-07-26T06:18:49.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.344+0000 7f0656600640 1 -- start start 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0656600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06500b1e80 0x7f0650155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0656600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0656600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f065015d5a0 0x7f065014d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0656600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f06500b5130 con 0x7f06500b23d0 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0656600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f06500b4fb0 con 0x7f06500b1e80 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0656600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f06500b52b0 con 0x7f065015d5a0 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:49.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38708/0 (socket says 172.21.15.163:38708) 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 -- 172.21.15.163:0/1725278705 learned_addr learned my addr 172.21.15.163:0/1725278705 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0655200640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06500b1e80 0x7f0650155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 -- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f065015d5a0 msgr2=0x7f065014d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f065015d5a0 0x7f065014d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 -- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06500b1e80 msgr2=0x7f0650155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06500b1e80 0x7f0650155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f064fe00640 1 -- 172.21.15.163:0/1725278705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0650157df0 con 0x7f06500b23d0 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.345+0000 7f0655200640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06500b1e80 0x7f0650155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.346+0000 7f064fe00640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b7740 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f063800c4c0 tx=0x7f063800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.346+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638016020 con 0x7f06500b23d0 2024-07-26T06:18:49.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.346+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0638012940 con 0x7f06500b23d0 2024-07-26T06:18:49.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.346+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0650151970 con 0x7f06500b23d0 2024-07-26T06:18:49.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.346+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638012d90 con 0x7f06500b23d0 2024-07-26T06:18:49.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.346+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0650152210 con 0x7f06500b23d0 2024-07-26T06:18:49.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.347+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0650151d60 con 0x7f06500b23d0 2024-07-26T06:18:49.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.348+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f0638019020 con 0x7f06500b23d0 2024-07-26T06:18:49.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.349+0000 7f064cc00640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f062c0763f0 0x7f062c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.349+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f0638097360 con 0x7f06500b23d0 2024-07-26T06:18:49.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.350+0000 7f0655200640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f062c0763f0 0x7f062c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:49.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.350+0000 7f0655200640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f062c0763f0 0x7f062c0788b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f064000a830 tx=0x7f0640008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:49.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.353+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0638020030 con 0x7f06500b23d0 2024-07-26T06:18:49.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.566+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f06500ce650 con 0x7f06500b23d0 2024-07-26T06:18:49.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.575+0000 7f064cc00640 1 -- 172.21.15.163:0/1725278705 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v37) ==== 71+0+0 (secure 0 0 0) 0x7f0638060db0 con 0x7f06500b23d0 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f062c0763f0 msgr2=0x7f062c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f062c0763f0 0x7f062c0788b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f064000a830 tx=0x7f0640008040 comp rx=0 tx=0).stop 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 msgr2=0x7f06500b7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b7740 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f063800c4c0 tx=0x7f063800c980 comp rx=0 tx=0).stop 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 shutdown_connections 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f062c0763f0 0x7f062c0788b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f065015d5a0 0x7f065014d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f06500b23d0 0x7f06500b7740 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 --2- 172.21.15.163:0/1725278705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06500b1e80 0x7f0650155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 >> 172.21.15.163:0/1725278705 conn(0x7f06500cfda0 msgr2=0x7f0650153ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:49.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 shutdown_connections 2024-07-26T06:18:49.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.578+0000 7f0656600640 1 -- 172.21.15.163:0/1725278705 wait complete. 2024-07-26T06:18:49.597 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-07-26T06:18:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.783+0000 7f419d800640 1 -- 172.21.15.163:0/3437835871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 msgr2=0x7f417c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.783+0000 7f419d800640 1 --2- 172.21.15.163:0/3437835871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f417c0051c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f419000c630 tx=0x7f417c005830 comp rx=0 tx=0).stop 2024-07-26T06:18:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.783+0000 7f419ce00640 1 -- 172.21.15.163:0/3437835871 reap_dead start 2024-07-26T06:18:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.783+0000 7f419d800640 1 -- 172.21.15.163:0/3437835871 shutdown_connections 2024-07-26T06:18:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.783+0000 7f419d800640 1 -- 172.21.15.163:0/3437835871 >> 172.21.15.163:0/3437835871 conn(0x7f41980b0e30 msgr2=0x7f41980b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.784+0000 7f419d800640 1 -- 172.21.15.163:0/3437835871 shutdown_connections 2024-07-26T06:18:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.784+0000 7f419d800640 1 -- 172.21.15.163:0/3437835871 wait complete. 2024-07-26T06:18:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.785+0000 7f419d800640 1 Processor -- start 2024-07-26T06:18:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.786+0000 7f419d800640 1 -- start start 2024-07-26T06:18:49.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.786+0000 7f419d800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f41980cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.786+0000 7f419d800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41980c58f0 0x7f41980c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f419d800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41980c5d70 0x7f41980c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f419d800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41980718c0 con 0x7f417c004d40 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f419d800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4198071740 con 0x7f41980c5d70 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f419d800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f41980b3e20 con 0x7f41980c58f0 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f4197e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f41980cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f4197e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f41980cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38720/0 (socket says 172.21.15.163:38720) 2024-07-26T06:18:49.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f4197e00640 1 -- 172.21.15.163:0/2740021008 learned_addr learned my addr 172.21.15.163:0/2740021008 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:49.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.787+0000 7f4197400640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41980c58f0 0x7f41980c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:49.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f4197e00640 1 -- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41980c58f0 msgr2=0x7f41980c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f4197e00640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41980c58f0 0x7f41980c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f4197e00640 1 -- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41980c5d70 msgr2=0x7f41980c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f4197e00640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41980c5d70 0x7f41980c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f4197e00640 1 -- 172.21.15.163:0/2740021008 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41980cbfe0 con 0x7f417c004d40 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f4197e00640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f41980cd080 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4188004e20 tx=0x7f4188001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188015020 con 0x7f417c004d40 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4188011960 con 0x7f417c004d40 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188001d40 con 0x7f417c004d40 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41980cc2c0 con 0x7f417c004d40 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.788+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41980cc880 con 0x7f417c004d40 2024-07-26T06:18:49.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.789+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41980b4380 con 0x7f417c004d40 2024-07-26T06:18:49.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.791+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f4188018020 con 0x7f417c004d40 2024-07-26T06:18:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.792+0000 7f418fe00640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4174076420 0x7f41740788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.792+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f4188098110 con 0x7f417c004d40 2024-07-26T06:18:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.792+0000 7f4197400640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4174076420 0x7f41740788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.793+0000 7f4197400640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4174076420 0x7f41740788e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41980cafc0 tx=0x7f4180005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:49.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:49.796+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f418801f020 con 0x7f417c004d40 2024-07-26T06:18:50.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.071+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7f41980ced60 con 0x7f417c004d40 2024-07-26T06:18:50.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.123+0000 7f418fe00640 1 -- 172.21.15.163:0/2740021008 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v280) ==== 146+0+0 (secure 0 0 0) 0x7f4188061a50 con 0x7f417c004d40 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4174076420 msgr2=0x7f41740788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4174076420 0x7f41740788e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41980cafc0 tx=0x7f4180005f90 comp rx=0 tx=0).stop 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 msgr2=0x7f41980cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f41980cd080 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4188004e20 tx=0x7f4188001630 comp rx=0 tx=0).stop 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 shutdown_connections 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4174076420 0x7f41740788e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41980c5d70 0x7f41980c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41980c58f0 0x7f41980c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 --2- 172.21.15.163:0/2740021008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f417c004d40 0x7f41980cd080 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.126+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 >> 172.21.15.163:0/2740021008 conn(0x7f41980b0e30 msgr2=0x7f419815dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:50.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.127+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 shutdown_connections 2024-07-26T06:18:50.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.127+0000 7f419d800640 1 -- 172.21.15.163:0/2740021008 wait complete. 2024-07-26T06:18:50.146 INFO:tasks.cephfs_test_runner:Test the creating vols, subvols subvolgroups fails when their names uses ... ok 2024-07-26T06:18:50.146 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-07-26T06:18:50.146 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:18:50.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:50.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 -- 172.21.15.163:0/3600997946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec1556f0 msgr2=0x7f05ec0cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 --2- 172.21.15.163:0/3600997946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec1556f0 0x7f05ec0cdd90 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0cea70 tx=0x7f05d402a560 comp rx=0 tx=0).stop 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 -- 172.21.15.163:0/3600997946 shutdown_connections 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 --2- 172.21.15.163:0/3600997946 >> v2:172.21.15.164:3300/0 conn(0x7f05ec071a70 0x7f05ec15ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 --2- 172.21.15.163:0/3600997946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05ec0ce2d0 0x7f05ec071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 --2- 172.21.15.163:0/3600997946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec1556f0 0x7f05ec0cdd90 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 -- 172.21.15.163:0/3600997946 >> 172.21.15.163:0/3600997946 conn(0x7f05ec0b0e30 msgr2=0x7f05ec0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 -- 172.21.15.163:0/3600997946 shutdown_connections 2024-07-26T06:18:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.346+0000 7f05f2400640 1 -- 172.21.15.163:0/3600997946 wait complete. 2024-07-26T06:18:50.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.348+0000 7f05f2400640 1 Processor -- start 2024-07-26T06:18:50.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.348+0000 7f05f2400640 1 -- start start 2024-07-26T06:18:50.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05f2400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05ec071a70 0x7f05ec14f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05f2400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 0x7f05ec156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05f2400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05ec1556f0 0x7f05ec1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05f2400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec15f2c0 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05f2400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec15f140 con 0x7f05ec071a70 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05f2400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05ec15f440 con 0x7f05ec1556f0 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05e3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 0x7f05ec156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05e3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 0x7f05ec156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38748/0 (socket says 172.21.15.163:38748) 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.349+0000 7f05e3e00640 1 -- 172.21.15.163:0/2456117844 learned_addr learned my addr 172.21.15.163:0/2456117844 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05f1000640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05ec071a70 0x7f05ec14f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e3e00640 1 -- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05ec1556f0 msgr2=0x7f05ec1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e3e00640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05ec1556f0 0x7f05ec1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e3e00640 1 -- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05ec071a70 msgr2=0x7f05ec14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e3e00640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05ec071a70 0x7f05ec14f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e3e00640 1 -- 172.21.15.163:0/2456117844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec14f930 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e3e00640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 0x7f05ec156a20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00b450 tx=0x7f05dc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:50.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc015020 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05dc011980 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc011dd0 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.350+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ec1574c0 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.351+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ec157a80 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.352+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ec15f980 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.353+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f05dc018020 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.354+0000 7f05e9600640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f05c8076420 0x7f05c80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.354+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f05dc0977d0 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.354+0000 7f05f1000640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f05c8076420 0x7f05c80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:50.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.355+0000 7f05f1000640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f05c8076420 0x7f05c80788e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05d4033000 tx=0x7f05d4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:50.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.358+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f05dc01f020 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.567+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05ec0b3cb0 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.569+0000 7f05e9600640 1 -- 172.21.15.163:0/2456117844 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+13179 (secure 0 0 0) 0x7f05dc00d040 con 0x7f05ec0ce2d0 2024-07-26T06:18:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:50.571 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":91,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:49.106940+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6834/3348438490":"2024-07-27T06:18:46.084770+0000","172.21.15.164:6816/3348438490":"2024-07-27T06:18:46.084770+0000","172.21.15.163:6834/2494990100":"2024-07-27T06:18:37.837948+0000","172.21.15.163:6832/2494990100":"2024-07-27T06:18:37.837948+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:50.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f05c8076420 msgr2=0x7f05c80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f05c8076420 0x7f05c80788e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05d4033000 tx=0x7f05d4004730 comp rx=0 tx=0).stop 2024-07-26T06:18:50.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 msgr2=0x7f05ec156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 0x7f05ec156a20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00b450 tx=0x7f05dc00b910 comp rx=0 tx=0).stop 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 shutdown_connections 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f05c8076420 0x7f05c80788e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05ec1556f0 0x7f05ec1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05ec0ce2d0 0x7f05ec156a20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 --2- 172.21.15.163:0/2456117844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05ec071a70 0x7f05ec14f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 >> 172.21.15.163:0/2456117844 conn(0x7f05ec0b0e30 msgr2=0x7f05ec15d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.572+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 shutdown_connections 2024-07-26T06:18:50.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.573+0000 7f05f2400640 1 -- 172.21.15.163:0/2456117844 wait complete. 2024-07-26T06:18:50.593 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:13:21.677101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:18:50.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T06:18:50.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df7200640 1 -- 172.21.15.163:0/1959073446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 msgr2=0x7f2ddc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df7200640 1 --2- 172.21.15.163:0/1959073446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2ddc003e50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c2e0 tx=0x7f2ddc005830 comp rx=0 tx=0).stop 2024-07-26T06:18:50.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df6800640 1 -- 172.21.15.163:0/1959073446 reap_dead start 2024-07-26T06:18:50.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df7200640 1 -- 172.21.15.163:0/1959073446 shutdown_connections 2024-07-26T06:18:50.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df7200640 1 -- 172.21.15.163:0/1959073446 >> 172.21.15.163:0/1959073446 conn(0x7f2df80b0e30 msgr2=0x7f2df80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:50.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df7200640 1 -- 172.21.15.163:0/1959073446 shutdown_connections 2024-07-26T06:18:50.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.800+0000 7f2df7200640 1 -- 172.21.15.163:0/1959073446 wait complete. 2024-07-26T06:18:50.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.802+0000 7f2df7200640 1 Processor -- start 2024-07-26T06:18:50.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.802+0000 7f2df7200640 1 -- start start 2024-07-26T06:18:50.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.802+0000 7f2df7200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2df80b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df7200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2df8149440 0x7f2df81498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df7200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2df8152910 0x7f2df8152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df7200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8164310 con 0x7f2ddc003a30 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df7200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8164190 con 0x7f2df8149440 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df7200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2df8164490 con 0x7f2df8152910 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2df80b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2df80b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38768/0 (socket says 172.21.15.163:38768) 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 -- 172.21.15.163:0/3844187981 learned_addr learned my addr 172.21.15.163:0/3844187981 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:50.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 -- 172.21.15.163:0/3844187981 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2df8152910 msgr2=0x7f2df8152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2df8152910 0x7f2df8152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 -- 172.21.15.163:0/3844187981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2df8149440 msgr2=0x7f2df81498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2df8149440 0x7f2df81498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.803+0000 7f2df5e00640 1 -- 172.21.15.163:0/3844187981 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df80cbc20 con 0x7f2ddc003a30 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.804+0000 7f2df5e00640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2df80b3980 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2de802afe0 tx=0x7f2de8001700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.804+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de80064d0 con 0x7f2ddc003a30 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.804+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de803e070 con 0x7f2ddc003a30 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.804+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de80039c0 con 0x7f2ddc003a30 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.804+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df80cbf00 con 0x7f2ddc003a30 2024-07-26T06:18:50.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.804+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df80cc7a0 con 0x7f2ddc003a30 2024-07-26T06:18:50.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.805+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df8071530 con 0x7f2ddc003a30 2024-07-26T06:18:50.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.806+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f2de8003b60 con 0x7f2ddc003a30 2024-07-26T06:18:50.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.807+0000 7f2dee000640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2dd8076420 0x7f2dd80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:50.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.807+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f2de802c020 con 0x7f2ddc003a30 2024-07-26T06:18:50.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.807+0000 7f2df5400640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2dd8076420 0x7f2dd80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:50.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.808+0000 7f2df5400640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2dd8076420 0x7f2dd80788e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2df80b0610 tx=0x7f2de4007970 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:50.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:50.812+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2de80815e0 con 0x7f2ddc003a30 2024-07-26T06:18:51.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.016+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2df80cc0d0 con 0x7f2ddc003a30 2024-07-26T06:18:51.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.017+0000 7f2dee000640 1 -- 172.21.15.163:0/3844187981 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f2de8080ee0 con 0x7f2ddc003a30 2024-07-26T06:18:51.018 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-07-26T06:18:51.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2dd8076420 msgr2=0x7f2dd80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:51.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2dd8076420 0x7f2dd80788e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2df80b0610 tx=0x7f2de4007970 comp rx=0 tx=0).stop 2024-07-26T06:18:51.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 msgr2=0x7f2df80b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:51.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2df80b3980 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2de802afe0 tx=0x7f2de8001700 comp rx=0 tx=0).stop 2024-07-26T06:18:51.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 shutdown_connections 2024-07-26T06:18:51.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2dd8076420 0x7f2dd80788e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2df8152910 0x7f2df8152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2df8149440 0x7f2df81498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 --2- 172.21.15.163:0/3844187981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ddc003a30 0x7f2df80b3980 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.020+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 >> 172.21.15.163:0/3844187981 conn(0x7f2df80b0e30 msgr2=0x7f2df80cdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:51.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.021+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 shutdown_connections 2024-07-26T06:18:51.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.021+0000 7f2df7200640 1 -- 172.21.15.163:0/3844187981 wait complete. 2024-07-26T06:18:51.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-07-26T06:18:51.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 -- 172.21.15.163:0/3010039762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 msgr2=0x7fa618005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:51.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 --2- 172.21.15.163:0/3010039762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa618005d90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa6200016c0 tx=0x7fa618006940 comp rx=0 tx=0).stop 2024-07-26T06:18:51.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 -- 172.21.15.163:0/3010039762 shutdown_connections 2024-07-26T06:18:51.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 --2- 172.21.15.163:0/3010039762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa618005d90 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 -- 172.21.15.163:0/3010039762 >> 172.21.15.163:0/3010039762 conn(0x7fa63006d420 msgr2=0x7fa63006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:51.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 -- 172.21.15.163:0/3010039762 shutdown_connections 2024-07-26T06:18:51.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.248+0000 7fa637c00640 1 -- 172.21.15.163:0/3010039762 wait complete. 2024-07-26T06:18:51.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.250+0000 7fa637c00640 1 Processor -- start 2024-07-26T06:18:51.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.250+0000 7fa637c00640 1 -- start start 2024-07-26T06:18:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa637c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa63014f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa637c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa630159970 0x7fa630159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa637c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa63015a330 0x7fa6301e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa637c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa63006bf40 con 0x7fa618005910 2024-07-26T06:18:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa637c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa6300b4580 con 0x7fa63015a330 2024-07-26T06:18:51.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa637c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa63006c0c0 con 0x7fa630159970 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa635200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa63014f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa635200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa63014f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38786/0 (socket says 172.21.15.163:38786) 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.251+0000 7fa635200640 1 -- 172.21.15.163:0/1925506604 learned_addr learned my addr 172.21.15.163:0/1925506604 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa63015a330 0x7fa6301e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa62fe00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa630159970 0x7fa630159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635200640 1 -- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa630159970 msgr2=0x7fa630159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635200640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa630159970 0x7fa630159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635200640 1 -- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa63015a330 msgr2=0x7fa6301e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635200640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa63015a330 0x7fa6301e70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635200640 1 -- 172.21.15.163:0/1925506604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63015a7b0 con 0x7fa618005910 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa635200640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa63014f320 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa62400c680 tx=0x7fa62400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:51.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624017070 con 0x7fa618005910 2024-07-26T06:18:51.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa624005360 con 0x7fa618005910 2024-07-26T06:18:51.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62400a8a0 con 0x7fa618005910 2024-07-26T06:18:51.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.252+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6301e6330 con 0x7fa618005910 2024-07-26T06:18:51.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.253+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6301e68f0 con 0x7fa618005910 2024-07-26T06:18:51.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.254+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63006c620 con 0x7fa618005910 2024-07-26T06:18:51.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.255+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fa62401a070 con 0x7fa618005910 2024-07-26T06:18:51.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.257+0000 7fa62cc00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa614076420 0x7fa6140788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:51.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.257+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7fa624096b60 con 0x7fa618005910 2024-07-26T06:18:51.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.257+0000 7fa62fe00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa614076420 0x7fa6140788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:51.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.258+0000 7fa62fe00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa614076420 0x7fa6140788e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa620032000 tx=0x7fa620001190 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:51.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.261+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa624060640 con 0x7fa618005910 2024-07-26T06:18:51.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:51.545+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7fa630156e60 con 0x7fa618005910 2024-07-26T06:18:52.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.137+0000 7fa62cc00640 1 -- 172.21.15.163:0/1925506604 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v282) ==== 152+0+0 (secure 0 0 0) 0x7fa62405ff40 con 0x7fa618005910 2024-07-26T06:18:52.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa614076420 msgr2=0x7fa6140788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa614076420 0x7fa6140788e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa620032000 tx=0x7fa620001190 comp rx=0 tx=0).stop 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 msgr2=0x7fa63014f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa63014f320 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa62400c680 tx=0x7fa62400cb40 comp rx=0 tx=0).stop 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 shutdown_connections 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa614076420 0x7fa6140788e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa63015a330 0x7fa6301e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa630159970 0x7fa630159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 --2- 172.21.15.163:0/1925506604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa618005910 0x7fa63014f320 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.141+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 >> 172.21.15.163:0/1925506604 conn(0x7fa63006d420 msgr2=0x7fa6300b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.142+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 shutdown_connections 2024-07-26T06:18:52.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.142+0000 7fa637c00640 1 -- 172.21.15.163:0/1925506604 wait complete. 2024-07-26T06:18:52.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-07-26T06:18:52.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 -- 172.21.15.163:0/198980107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 msgr2=0x7f95cc0ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 --2- 172.21.15.163:0/198980107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc0ad8e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f95cc06f6e0 tx=0x7f95bc02a560 comp rx=0 tx=0).stop 2024-07-26T06:18:52.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 -- 172.21.15.163:0/198980107 shutdown_connections 2024-07-26T06:18:52.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 --2- 172.21.15.163:0/198980107 >> v2:172.21.15.164:3300/0 conn(0x7f95cc0cc450 0x7f95cc0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 --2- 172.21.15.163:0/198980107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc0ad8e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 -- 172.21.15.163:0/198980107 >> 172.21.15.163:0/198980107 conn(0x7f95cc0b0e30 msgr2=0x7f95cc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:52.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 -- 172.21.15.163:0/198980107 shutdown_connections 2024-07-26T06:18:52.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.384+0000 7f95d2400640 1 -- 172.21.15.163:0/198980107 wait complete. 2024-07-26T06:18:52.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.386+0000 7f95d2400640 1 Processor -- start 2024-07-26T06:18:52.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.386+0000 7f95d2400640 1 -- start start 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d2400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95cc0ade20 0x7f95cc15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d2400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95cc0cc450 0x7f95cc13e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d2400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d2400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc0b4020 con 0x7f95cc146be0 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d2400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc0b3ea0 con 0x7f95cc0ade20 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d2400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f95cc0b41a0 con 0x7f95cc0cc450 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d1a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d1a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc072ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38796/0 (socket says 172.21.15.163:38796) 2024-07-26T06:18:52.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.387+0000 7f95d1a00640 1 -- 172.21.15.163:0/2227516069 learned_addr learned my addr 172.21.15.163:0/2227516069 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1000640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95cc0ade20 0x7f95cc15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95c3e00640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95cc0cc450 0x7f95cc13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1a00640 1 -- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95cc0cc450 msgr2=0x7f95cc13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1a00640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95cc0cc450 0x7f95cc13e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1a00640 1 -- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95cc0ade20 msgr2=0x7f95cc15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1a00640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95cc0ade20 0x7f95cc15cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1a00640 1 -- 172.21.15.163:0/2227516069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95cc13f0a0 con 0x7f95cc146be0 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95d1a00640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc072ff0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f95c400b970 tx=0x7f95c400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:52.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.388+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c4019070 con 0x7f95cc146be0 2024-07-26T06:18:52.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.389+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95c400cc50 con 0x7f95cc146be0 2024-07-26T06:18:52.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.389+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c40146a0 con 0x7f95cc146be0 2024-07-26T06:18:52.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.389+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95cc13f380 con 0x7f95cc146be0 2024-07-26T06:18:52.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.389+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95cc13f940 con 0x7f95cc146be0 2024-07-26T06:18:52.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.390+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95cc0b4780 con 0x7f95cc146be0 2024-07-26T06:18:52.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.391+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f95c401c070 con 0x7f95cc146be0 2024-07-26T06:18:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.392+0000 7f95c9600640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f95a80763f0 0x7f95a80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.392+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f95c4097460 con 0x7f95cc146be0 2024-07-26T06:18:52.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.392+0000 7f95d1000640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f95a80763f0 0x7f95a80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.393+0000 7f95d1000640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f95a80763f0 0x7f95a80788b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f95bc033000 tx=0x7f95bc004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:52.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.396+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95c4060f40 con 0x7f95cc146be0 2024-07-26T06:18:52.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.598+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f95cc1f4120 con 0x7f95cc146be0 2024-07-26T06:18:52.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.600+0000 7f95c9600640 1 -- 172.21.15.163:0/2227516069 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+1998 (secure 0 0 0) 0x7f95c4060840 con 0x7f95cc146be0 2024-07-26T06:18:52.600 INFO:teuthology.orchestra.run.smithi163.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-07-26T06:18:52.600 INFO:teuthology.orchestra.run.smithi163.stdout:mon advanced mon_op_complaint_time 120 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:mon dev mon_warn_on_pool_no_app false 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd advanced osd_op_complaint_time 180.000000 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 5439.226762 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 3901.149445 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5414.054049 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4219.540367 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4287.026771 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4403.390021 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 3932.844298 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4626.659828 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds 20/20 2024-07-26T06:18:52.601 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_mds_balancer 20/20 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced debug_ms 1/1 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_frag true 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_debug_scatterstat true 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced mds_op_complaint_time 180.000000 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds dev mds_verify_scatter true 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced osd_op_complaint_time 180.000000 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_mon_op_timeout 900 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:mds advanced rados_osd_op_timeout 900 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced client_mount_timeout 600 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_client 20/20 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced debug_ms 1/1 2024-07-26T06:18:52.602 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_mon_op_timeout 900 2024-07-26T06:18:52.603 INFO:teuthology.orchestra.run.smithi163.stdout:client advanced rados_osd_op_timeout 900 2024-07-26T06:18:52.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f95a80763f0 msgr2=0x7f95a80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f95a80763f0 0x7f95a80788b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f95bc033000 tx=0x7f95bc004730 comp rx=0 tx=0).stop 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 msgr2=0x7f95cc072ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc072ff0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f95c400b970 tx=0x7f95c400be30 comp rx=0 tx=0).stop 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 shutdown_connections 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f95a80763f0 0x7f95a80788b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f95cc146be0 0x7f95cc072ff0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95cc0cc450 0x7f95cc13e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 --2- 172.21.15.163:0/2227516069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95cc0ade20 0x7f95cc15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.602+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 >> 172.21.15.163:0/2227516069 conn(0x7f95cc0b0e30 msgr2=0x7f95cc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.603+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 shutdown_connections 2024-07-26T06:18:52.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.603+0000 7f95d2400640 1 -- 172.21.15.163:0/2227516069 wait complete. 2024-07-26T06:18:52.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 -- 172.21.15.163:0/1909887312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 msgr2=0x7fa68c0b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 --2- 172.21.15.163:0/1909887312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c0b64c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fa68c0690b0 tx=0x7fa67402a4b0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 -- 172.21.15.163:0/1909887312 shutdown_connections 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 --2- 172.21.15.163:0/1909887312 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c0b64c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 --2- 172.21.15.163:0/1909887312 >> v2:172.21.15.164:3300/0 conn(0x7fa68c15d5a0 0x7fa68c0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 -- 172.21.15.163:0/1909887312 >> 172.21.15.163:0/1909887312 conn(0x7fa68c0cfda0 msgr2=0x7fa68c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:52.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 -- 172.21.15.163:0/1909887312 shutdown_connections 2024-07-26T06:18:52.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.823+0000 7fa691200640 1 -- 172.21.15.163:0/1909887312 wait complete. 2024-07-26T06:18:52.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.825+0000 7fa691200640 1 Processor -- start 2024-07-26T06:18:52.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.825+0000 7fa691200640 1 -- start start 2024-07-26T06:18:52.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa691200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa68c0b1e80 0x7fa68c151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa691200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa691200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa68c15d5a0 0x7fa68c1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa691200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c0b5130 con 0x7fa68c0ce830 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa691200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c0b4fb0 con 0x7fa68c0b1e80 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa691200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa68c0b52b0 con 0x7fa68c15d5a0 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa68aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa68aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c152c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38812/0 (socket says 172.21.15.163:38812) 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa68aa00640 1 -- 172.21.15.163:0/544221851 learned_addr learned my addr 172.21.15.163:0/544221851 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa68b400640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa68c0b1e80 0x7fa68c151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa68aa00640 1 -- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa68c15d5a0 msgr2=0x7fa68c1534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.826+0000 7fa68be00640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa68c15d5a0 0x7fa68c1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa68aa00640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa68c15d5a0 0x7fa68c1534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa68aa00640 1 -- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa68c0b1e80 msgr2=0x7fa68c151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa68aa00640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa68c0b1e80 0x7fa68c151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa68aa00640 1 -- 172.21.15.163:0/544221851 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68c15b0a0 con 0x7fa68c0ce830 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa68b400640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa68c0b1e80 0x7fa68c151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa68aa00640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c152c70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa674009780 tx=0x7fa674032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674040070 con 0x7fa68c0ce830 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa674005310 con 0x7fa68c0ce830 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa68c15b380 con 0x7fa68c0ce830 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6740047a0 con 0x7fa68c0ce830 2024-07-26T06:18:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.827+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa68c155c10 con 0x7fa68c0ce830 2024-07-26T06:18:52.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.829+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa68c074680 con 0x7fa68c0ce830 2024-07-26T06:18:52.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.830+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fa674039040 con 0x7fa68c0ce830 2024-07-26T06:18:52.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.831+0000 7fa683400640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa668076420 0x7fa6680788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:52.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.831+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7fa6740b8b70 con 0x7fa68c0ce830 2024-07-26T06:18:52.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.831+0000 7fa68b400640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa668076420 0x7fa6680788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:52.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.832+0000 7fa68b400640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa668076420 0x7fa6680788e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa67c005fd0 tx=0x7fa67c009340 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:52.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:52.835+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa6740824b0 con 0x7fa68c0ce830 2024-07-26T06:18:53.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.042+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fa68c15b880 con 0x7fa68c0ce830 2024-07-26T06:18:53.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.049+0000 7fa683400640 1 -- 172.21.15.163:0/544221851 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7fa674081db0 con 0x7fa68c0ce830 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.052+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa668076420 msgr2=0x7fa6680788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.052+0000 7fa691200640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa668076420 0x7fa6680788e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa67c005fd0 tx=0x7fa67c009340 comp rx=0 tx=0).stop 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.052+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 msgr2=0x7fa68c152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.052+0000 7fa691200640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c152c70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa674009780 tx=0x7fa674032c10 comp rx=0 tx=0).stop 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 shutdown_connections 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa668076420 0x7fa6680788e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa68c15d5a0 0x7fa68c1534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa68c0ce830 0x7fa68c152c70 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 --2- 172.21.15.163:0/544221851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa68c0b1e80 0x7fa68c151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 >> 172.21.15.163:0/544221851 conn(0x7fa68c0cfda0 msgr2=0x7fa68c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:53.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 shutdown_connections 2024-07-26T06:18:53.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.053+0000 7fa691200640 1 -- 172.21.15.163:0/544221851 wait complete. 2024-07-26T06:18:53.074 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:18:53.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:53.186 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:18:53.322 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:18:53.322 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:53.322 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:53.381 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:53.381 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:53.381 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:53.440 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:53.440 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:18:53.440 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:53.497 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:18:53.498 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:18:53.498 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:18:53.555 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:18:53.556 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-07-26T06:18:53.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:18:53.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 -- 172.21.15.163:0/3829076494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680b1e80 msgr2=0x7f3f680b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:53.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 --2- 172.21.15.163:0/3829076494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680b1e80 0x7f3f680b6a00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3f680690b0 tx=0x7f3f6002a8f0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 -- 172.21.15.163:0/3829076494 shutdown_connections 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 --2- 172.21.15.163:0/3829076494 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680b1e80 0x7f3f680b6a00 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 --2- 172.21.15.163:0/3829076494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f680ce830 0x7f3f680b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 --2- 172.21.15.163:0/3829076494 >> v2:172.21.15.164:3300/0 conn(0x7f3f6815d5a0 0x7f3f680ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 -- 172.21.15.163:0/3829076494 >> 172.21.15.163:0/3829076494 conn(0x7f3f680cfda0 msgr2=0x7f3f680d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 -- 172.21.15.163:0/3829076494 shutdown_connections 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.769+0000 7f3f6d200640 1 -- 172.21.15.163:0/3829076494 wait complete. 2024-07-26T06:18:53.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.770+0000 7f3f6d200640 1 Processor -- start 2024-07-26T06:18:53.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.771+0000 7f3f6d200640 1 -- start start 2024-07-26T06:18:53.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f6d200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f680b1e80 0x7f3f68156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:53.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f6d200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 0x7f3f680c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:53.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f6d200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f6815d5a0 0x7f3f6815e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:53.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f6d200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680b4da0 con 0x7f3f680ce830 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f6d200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680b2660 con 0x7f3f680b1e80 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f6d200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f680b4f20 con 0x7f3f6815d5a0 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 0x7f3f680c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 0x7f3f680c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38830/0 (socket says 172.21.15.163:38830) 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 -- 172.21.15.163:0/881479434 learned_addr learned my addr 172.21.15.163:0/881479434 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f67400640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f680b1e80 0x7f3f68156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f67e00640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f6815d5a0 0x7f3f6815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 -- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f6815d5a0 msgr2=0x7f3f6815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f6815d5a0 0x7f3f6815e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 -- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f680b1e80 msgr2=0x7f3f68156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f680b1e80 0x7f3f68156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f66a00640 1 -- 172.21.15.163:0/881479434 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f681f43e0 con 0x7f3f680ce830 2024-07-26T06:18:53.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.772+0000 7f3f67400640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f680b1e80 0x7f3f68156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:53.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.773+0000 7f3f66a00640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 0x7f3f680c7e60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c940 tx=0x7f3f5400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:53.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.773+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54004230 con 0x7f3f680ce830 2024-07-26T06:18:53.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.773+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f540079b0 con 0x7f3f680ce830 2024-07-26T06:18:53.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.773+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f681f46c0 con 0x7f3f680ce830 2024-07-26T06:18:53.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.773+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54007e00 con 0x7f3f680ce830 2024-07-26T06:18:53.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.773+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f680b79d0 con 0x7f3f680ce830 2024-07-26T06:18:53.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.774+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f680ce2f0 con 0x7f3f680ce830 2024-07-26T06:18:53.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.775+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f3f540035f0 con 0x7f3f680ce830 2024-07-26T06:18:53.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.776+0000 7f3f5f400640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3f48076420 0x7f3f480788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:53.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.777+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f3f54097880 con 0x7f3f680ce830 2024-07-26T06:18:53.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.777+0000 7f3f67400640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3f48076420 0x7f3f480788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:53.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.777+0000 7f3f67400640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3f48076420 0x7f3f480788e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3f580058c0 tx=0x7f3f58005c90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:53.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:53.781+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3f54014070 con 0x7f3f680ce830 2024-07-26T06:18:54.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.024+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f68158270 con 0x7f3f680ce830 2024-07-26T06:18:54.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.025+0000 7f3f5f400640 1 -- 172.21.15.163:0/881479434 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+3920 (secure 0 0 0) 0x7f3f540611c0 con 0x7f3f680ce830 2024-07-26T06:18:54.026 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:54.026 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":63,"btime":"2024-07-26T06:18:47:089793+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/1048193975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":1048193975},{"type":"v1","addr":"172.21.15.163:6835","nonce":1048193975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/2861492560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":2861492560},{"type":"v1","addr":"172.21.15.164:6834","nonce":2861492560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5476,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/285896864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":285896864},{"type":"v1","addr":"172.21.15.164:6835","nonce":285896864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6720,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/3979720531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3979720531},{"type":"v1","addr":"172.21.15.163:6834","nonce":3979720531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[]} 2024-07-26T06:18:54.027 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 63 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3f48076420 msgr2=0x7f3f480788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3f48076420 0x7f3f480788e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3f580058c0 tx=0x7f3f58005c90 comp rx=0 tx=0).stop 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 msgr2=0x7f3f680c7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 0x7f3f680c7e60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c940 tx=0x7f3f5400ce00 comp rx=0 tx=0).stop 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 shutdown_connections 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3f48076420 0x7f3f480788e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f6815d5a0 0x7f3f6815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f680ce830 0x7f3f680c7e60 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 --2- 172.21.15.163:0/881479434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f680b1e80 0x7f3f68156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 >> 172.21.15.163:0/881479434 conn(0x7f3f680cfda0 msgr2=0x7f3f680d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:54.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 shutdown_connections 2024-07-26T06:18:54.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.028+0000 7f3f6d200640 1 -- 172.21.15.163:0/881479434 wait complete. 2024-07-26T06:18:54.047 INFO:tasks.ceph.mds.a:Restarting daemon 2024-07-26T06:18:54.047 INFO:tasks.ceph.mds.a:Stopping old one... 2024-07-26T06:18:54.047 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:18:54.047 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:54.047 INFO:tasks.ceph.mds.c:Restarting daemon 2024-07-26T06:18:54.048 INFO:tasks.ceph.mds.c:Stopping old one... 2024-07-26T06:18:54.048 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:18:54.048 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:54.048 INFO:tasks.ceph.mds.b:Restarting daemon 2024-07-26T06:18:54.048 INFO:tasks.ceph.mds.b:Stopping old one... 2024-07-26T06:18:54.048 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:18:54.048 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:54.049 INFO:tasks.ceph.mds.d:Restarting daemon 2024-07-26T06:18:54.049 INFO:tasks.ceph.mds.d:Stopping old one... 2024-07-26T06:18:54.049 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:18:54.049 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:18:54.061 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:18:54.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-07-26T06:18:54.065 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:18:54.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-07-26T06:18:54.074 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:18:54.074 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-07-26T06:18:54.078 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:18:54.079 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-07-26T06:18:54.111 INFO:tasks.ceph.mds.a:Started 2024-07-26T06:18:54.111 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:54.117 INFO:tasks.ceph.mds.c:Started 2024-07-26T06:18:54.117 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:54.126 INFO:tasks.ceph.mds.b:Started 2024-07-26T06:18:54.126 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:54.132 INFO:tasks.ceph.mds.d:Started 2024-07-26T06:18:54.133 DEBUG:teuthology.parallel:result is None 2024-07-26T06:18:54.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-07-26T06:18:54.255 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:18:54.255 INFO:tasks.ceph.mds.a.smithi163.stdout:starting mds.a at 2024-07-26T06:18:54.278 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-07-26T06:18:54.279 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-07-26T06:18:54.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.307+0000 7f59da200640 1 -- 172.21.15.163:0/3738704212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d40b23d0 msgr2=0x7f59d40b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.307+0000 7f59da200640 1 --2- 172.21.15.163:0/3738704212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d40b23d0 0x7f59d40b6bd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f59cc005fd0 tx=0x7f59cc02a820 comp rx=0 tx=0).stop 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.308+0000 7f59da200640 1 -- 172.21.15.163:0/3738704212 shutdown_connections 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.308+0000 7f59da200640 1 --2- 172.21.15.163:0/3738704212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d40b23d0 0x7f59d40b6bd0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.308+0000 7f59da200640 1 -- 172.21.15.163:0/3738704212 >> 172.21.15.163:0/3738704212 conn(0x7f59d40cfda0 msgr2=0x7f59d40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.308+0000 7f59da200640 1 -- 172.21.15.163:0/3738704212 shutdown_connections 2024-07-26T06:18:54.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.308+0000 7f59da200640 1 -- 172.21.15.163:0/3738704212 wait complete. 2024-07-26T06:18:54.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.309+0000 7f59da200640 1 Processor -- start 2024-07-26T06:18:54.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.309+0000 7f59da200640 1 -- start start 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59da200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d40b1e80 0x7f59d414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59da200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d40b23d0 0x7f59d40ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59da200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 0x7f59d40b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59da200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59d40b5130 con 0x7f59d415d5a0 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59da200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f59d40b4fb0 con 0x7f59d40b1e80 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59da200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f59d40b52b0 con 0x7f59d40b23d0 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.310+0000 7f59d9800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 0x7f59d40b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 0x7f59d40b7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:38880/0 (socket says 172.21.15.163:38880) 2024-07-26T06:18:54.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 -- 172.21.15.163:0/994170203 learned_addr learned my addr 172.21.15.163:0/994170203 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d40b23d0 0x7f59d40ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d8e00640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d40b1e80 0x7f59d414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 -- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d40b23d0 msgr2=0x7f59d40ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d40b23d0 0x7f59d40ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 -- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d40b1e80 msgr2=0x7f59d414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d40b1e80 0x7f59d414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 -- 172.21.15.163:0/994170203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d40cbfd0 con 0x7f59d415d5a0 2024-07-26T06:18:54.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.311+0000 7f59d9800640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 0x7f59d40b7a30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f59cc006010 tx=0x7f59cc003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.312+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc037040 con 0x7f59d415d5a0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.312+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59cc042070 con 0x7f59d415d5a0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.312+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc004530 con 0x7f59d415d5a0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.312+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d40cc2b0 con 0x7f59d415d5a0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.312+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d414dae0 con 0x7f59d415d5a0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.313+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d40cf5a0 con 0x7f59d415d5a0 2024-07-26T06:18:54.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.314+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f59cc0046d0 con 0x7f59d415d5a0 2024-07-26T06:18:54.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.315+0000 7f59d0c00640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f59b40763f0 0x7f59b40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:54.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.315+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5029+0+0 (secure 0 0 0) 0x7f59cc0b8630 con 0x7f59d415d5a0 2024-07-26T06:18:54.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.315+0000 7f59d8e00640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f59b40763f0 0x7f59b40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:54.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.316+0000 7f59d8e00640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f59b40763f0 0x7f59b40788b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59c8007fd0 tx=0x7f59c8001370 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:54.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.319+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f59d40cf5a0 con 0x7f59d415d5a0 2024-07-26T06:18:54.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:54.523+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f59d40b5810 con 0x7f59d415d5a0 2024-07-26T06:18:55.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.176+0000 7f59d0c00640 1 -- 172.21.15.163:0/994170203 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v93) ==== 97+0+0 (secure 0 0 0) 0x7f59cc0819c0 con 0x7f59d415d5a0 2024-07-26T06:18:55.178 INFO:teuthology.orchestra.run.smithi163.stderr: removed all blocklist entries 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.181+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f59b40763f0 msgr2=0x7f59b40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.181+0000 7f59da200640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f59b40763f0 0x7f59b40788b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59c8007fd0 tx=0x7f59c8001370 comp rx=0 tx=0).stop 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.181+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 msgr2=0x7f59d40b7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.181+0000 7f59da200640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 0x7f59d40b7a30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f59cc006010 tx=0x7f59cc003f60 comp rx=0 tx=0).stop 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 shutdown_connections 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f59b40763f0 0x7f59b40788b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d415d5a0 0x7f59d40b7a30 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d40b23d0 0x7f59d40ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 --2- 172.21.15.163:0/994170203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d40b1e80 0x7f59d414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 >> 172.21.15.163:0/994170203 conn(0x7f59d40cfda0 msgr2=0x7f59d40d1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 shutdown_connections 2024-07-26T06:18:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.182+0000 7f59da200640 1 -- 172.21.15.163:0/994170203 wait complete. 2024-07-26T06:18:55.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-07-26T06:18:55.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 -- 172.21.15.163:0/1495014885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 msgr2=0x7fa8140b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 --2- 172.21.15.163:0/1495014885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa8140b4bd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa81415ebd0 tx=0x7fa80802a4d0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 -- 172.21.15.163:0/1495014885 shutdown_connections 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 --2- 172.21.15.163:0/1495014885 >> v2:172.21.15.164:3300/0 conn(0x7fa81415db10 0x7fa81415dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 --2- 172.21.15.163:0/1495014885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8140ae870 0x7fa8140b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 --2- 172.21.15.163:0/1495014885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa8140b4bd0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 -- 172.21.15.163:0/1495014885 >> 172.21.15.163:0/1495014885 conn(0x7fa8140b0e30 msgr2=0x7fa8140b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:55.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 -- 172.21.15.163:0/1495014885 shutdown_connections 2024-07-26T06:18:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.404+0000 7fa81ae00640 1 -- 172.21.15.163:0/1495014885 wait complete. 2024-07-26T06:18:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.406+0000 7fa81ae00640 1 Processor -- start 2024-07-26T06:18:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.406+0000 7fa81ae00640 1 -- start start 2024-07-26T06:18:55.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa81ae00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8140ae870 0x7fa81414f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa81ae00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa81413d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa81ae00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa81415db10 0x7fa81414d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa81ae00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa8140cb870 con 0x7fa8140b3bd0 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa81ae00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa814073340 con 0x7fa8140ae870 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa81ae00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa8140cb9f0 con 0x7fa81415db10 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa819000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa81413d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.407+0000 7fa819000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa81413d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35708/0 (socket says 172.21.15.163:35708) 2024-07-26T06:18:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 -- 172.21.15.163:0/619164714 learned_addr learned my addr 172.21.15.163:0/619164714 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 -- 172.21.15.163:0/619164714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa81415db10 msgr2=0x7fa81414d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa81415db10 0x7fa81414d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 -- 172.21.15.163:0/619164714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8140ae870 msgr2=0x7fa81414f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8140ae870 0x7fa81414f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 -- 172.21.15.163:0/619164714 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81414e020 con 0x7fa8140b3bd0 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa819000640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa81413d8a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa80400b4d0 tx=0x7fa80400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804014070 con 0x7fa8140b3bd0 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa804010960 con 0x7fa8140b3bd0 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804010db0 con 0x7fa8140b3bd0 2024-07-26T06:18:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.408+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81414e300 con 0x7fa8140b3bd0 2024-07-26T06:18:55.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.409+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81414e8c0 con 0x7fa8140b3bd0 2024-07-26T06:18:55.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.410+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8140b3fb0 con 0x7fa8140b3bd0 2024-07-26T06:18:55.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.411+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fa804017070 con 0x7fa8140b3bd0 2024-07-26T06:18:55.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.413+0000 7fa811600640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa7f80763f0 0x7fa7f80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.413+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7fa804097440 con 0x7fa8140b3bd0 2024-07-26T06:18:55.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.413+0000 7fa819a00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa7f80763f0 0x7fa7f80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:55.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.414+0000 7fa819a00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa7f80763f0 0x7fa7f80788b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa808033000 tx=0x7fa8080046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:55.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.417+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa804060fd0 con 0x7fa8140b3bd0 2024-07-26T06:18:55.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.680+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa81414c7f0 con 0x7fa8140b3bd0 2024-07-26T06:18:55.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.681+0000 7fa811600640 1 -- 172.21.15.163:0/619164714 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7fa8040608d0 con 0x7fa8140b3bd0 2024-07-26T06:18:55.682 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:55.682 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_dump": [ 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.a", 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmiSuZCxAA8M1WC9Xtxvy7BCPDOLsptA==", 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.b", 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmQksMKBAAYxpmgRMUyI5WnoxZl5LQLg==", 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.c", 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNm3oKlGRAAPTmsca9LXmVedug8NOCOLw==", 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.685 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mds.d", 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmkU8hNhAAbEoUn/LD3o1rUEW21cCSlg==", 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.686 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.0", 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBRPqNmfUHGNBAAabt0SAhKhy0AIyw232/gOA==", 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.1", 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBVPqNmJeMdExAAWvYwTL11DY8PkUiNl66FjA==", 2024-07-26T06:18:55.687 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.2", 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBYPqNmskwSHhAAz8rsltTz1bXR9BzVXWbqbA==", 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.688 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.3", 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBbPqNmLS0/OBAAavnIJLPrv8j0ClizkInfIg==", 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.689 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.4", 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBiPqNmkHQwCxAA0vf1ysM9iX1vv1R7O3kjHA==", 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.5", 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBlPqNmsdllFhAA8THpG/rZAbrsYnlShU2V6g==", 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.690 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.6", 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBoPqNmbi+NIRAA2uppkMEIYEYo09khcFBNsQ==", 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "osd.7", 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBrPqNmvl8LKxAAHEejeiUnQ8ADNh2t066VwA==", 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow profile osd", 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile osd", 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.0", 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew==", 2024-07-26T06:18:55.692 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow r", 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.1", 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmbfVmGRAAD2AAnEXI8LtNJNgHW+OPKQ==", 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:18:55.693 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.2", 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmiahmDxAA+tQT8458FHHW5iIGKveKXw==", 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:18:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.3", 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBOPqNmABWqIhAAtjiecxjn+B3f1foz9/aJVw==", 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow", 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow r", 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow rw", 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow rwx" 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "client.admin", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm+ibKBxAA1ZBGvVy5rNIzMWwOFwFJFw==", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr": "allow *", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow *", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.x", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBNPqNmpL2JAhAAKyRA5Yemhrtz5JvtYtsnGQ==", 2024-07-26T06:18:55.696 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "entity": "mgr.y", 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "AQBMPqNm1mnVMxAA+kKW2rQKJCuvRHv9zrn3gQ==", 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": { 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": "allow *", 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "mon": "allow profile mgr", 2024-07-26T06:18:55.697 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": "allow *" 2024-07-26T06:18:55.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:55.698 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:18:55.698 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:18:55.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa7f80763f0 msgr2=0x7fa7f80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa7f80763f0 0x7fa7f80788b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa808033000 tx=0x7fa8080046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 msgr2=0x7fa81413d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa81413d8a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa80400b4d0 tx=0x7fa80400b990 comp rx=0 tx=0).stop 2024-07-26T06:18:55.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 shutdown_connections 2024-07-26T06:18:55.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa7f80763f0 0x7fa7f80788b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa81415db10 0x7fa81414d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8140b3bd0 0x7fa81413d8a0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 --2- 172.21.15.163:0/619164714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8140ae870 0x7fa81414f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 >> 172.21.15.163:0/619164714 conn(0x7fa8140b0e30 msgr2=0x7fa814074250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:55.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.684+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 shutdown_connections 2024-07-26T06:18:55.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.685+0000 7fa81ae00640 1 -- 172.21.15.163:0/619164714 wait complete. 2024-07-26T06:18:55.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-07-26T06:18:55.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.906+0000 7f6ce4e00640 1 -- 172.21.15.163:0/1589017884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce015d5a0 msgr2=0x7f6ce00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.906+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/1589017884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce015d5a0 0x7f6ce00ce2f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009830 tx=0x7f6cd002a480 comp rx=0 tx=0).stop 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.906+0000 7f6ce4e00640 1 -- 172.21.15.163:0/1589017884 shutdown_connections 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.906+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/1589017884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ce00b1e80 0x7f6ce00b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.906+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/1589017884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce015d5a0 0x7f6ce00ce2f0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.906+0000 7f6ce4e00640 1 -- 172.21.15.163:0/1589017884 >> 172.21.15.163:0/1589017884 conn(0x7f6ce00cfda0 msgr2=0x7f6ce00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.907+0000 7f6ce4e00640 1 -- 172.21.15.163:0/1589017884 shutdown_connections 2024-07-26T06:18:55.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.907+0000 7f6ce4e00640 1 -- 172.21.15.163:0/1589017884 wait complete. 2024-07-26T06:18:55.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.909+0000 7f6ce4e00640 1 Processor -- start 2024-07-26T06:18:55.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.909+0000 7f6ce4e00640 1 -- start start 2024-07-26T06:18:55.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6ce4e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ce00b1e80 0x7f6ce00c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6ce4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 0x7f6ce00ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6ce4e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ce015d5a0 0x7f6ce00ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6ce4e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce00b5130 con 0x7f6ce00ce830 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6ce4e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce00b4fb0 con 0x7f6ce015d5a0 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6ce4e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce00b52b0 con 0x7f6ce00b1e80 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6cdea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 0x7f6ce00ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.910+0000 7f6cdea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 0x7f6ce00ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35732/0 (socket says 172.21.15.163:35732) 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 -- 172.21.15.163:0/4036002472 learned_addr learned my addr 172.21.15.163:0/4036002472 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdf400640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ce00b1e80 0x7f6ce00c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:55.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdfe00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ce015d5a0 0x7f6ce00ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 -- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ce00b1e80 msgr2=0x7f6ce00c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ce00b1e80 0x7f6ce00c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 -- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ce015d5a0 msgr2=0x7f6ce00ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ce015d5a0 0x7f6ce00ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 -- 172.21.15.163:0/4036002472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce00c8f90 con 0x7f6ce00ce830 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.911+0000 7f6cdea00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 0x7f6ce00ca3f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00c490 tx=0x7f6ccc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:55.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.912+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc016020 con 0x7f6ce00ce830 2024-07-26T06:18:55.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.912+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ccc012940 con 0x7f6ce00ce830 2024-07-26T06:18:55.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.912+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc012d90 con 0x7f6ce00ce830 2024-07-26T06:18:55.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.912+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce00c9270 con 0x7f6ce00ce830 2024-07-26T06:18:55.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.912+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce00c9830 con 0x7f6ce00ce830 2024-07-26T06:18:55.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.913+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce00b57d0 con 0x7f6ce00ce830 2024-07-26T06:18:55.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.914+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f6ccc019020 con 0x7f6ce00ce830 2024-07-26T06:18:55.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.915+0000 7f6cd7400640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cc00763f0 0x7f6cc00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:55.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.915+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7f6ccc097230 con 0x7f6ce00ce830 2024-07-26T06:18:55.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.915+0000 7f6cdf400640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cc00763f0 0x7f6cc00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:55.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.916+0000 7f6cdf400640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cc00763f0 0x7f6cc00788b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0034000 tx=0x7f6cd0004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:55.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:55.919+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6ccc020030 con 0x7f6ce00ce830 2024-07-26T06:18:56.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.130+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6ce015cb40 con 0x7f6ce00ce830 2024-07-26T06:18:56.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.131+0000 7f6cd7400640 1 -- 172.21.15.163:0/4036002472 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) ==== 84+0+65 (secure 0 0 0) 0x7f6ccc060e60 con 0x7f6ce00ce830 2024-07-26T06:18:56.132 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:56.132 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:18:56.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:18:56.132 INFO:teuthology.orchestra.run.smithi163.stdout: "poolnum": 1, 2024-07-26T06:18:56.132 INFO:teuthology.orchestra.run.smithi163.stdout: "poolname": ".mgr" 2024-07-26T06:18:56.133 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:18:56.133 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:18:56.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cc00763f0 msgr2=0x7f6cc00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:56.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cc00763f0 0x7f6cc00788b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0034000 tx=0x7f6cd0004730 comp rx=0 tx=0).stop 2024-07-26T06:18:56.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 msgr2=0x7f6ce00ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:56.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 0x7f6ce00ca3f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00c490 tx=0x7f6ccc00c950 comp rx=0 tx=0).stop 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 shutdown_connections 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f6cc00763f0 0x7f6cc00788b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ce015d5a0 0x7f6ce00ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ce00ce830 0x7f6ce00ca3f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 --2- 172.21.15.163:0/4036002472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ce00b1e80 0x7f6ce00c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 >> 172.21.15.163:0/4036002472 conn(0x7f6ce00cfda0 msgr2=0x7f6ce00b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.134+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 shutdown_connections 2024-07-26T06:18:56.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.135+0000 7f6ce4e00640 1 -- 172.21.15.163:0/4036002472 wait complete. 2024-07-26T06:18:56.153 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:18:56.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:18:56.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 -- 172.21.15.163:0/3548729940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d240cbad0 msgr2=0x7f4d240cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 --2- 172.21.15.163:0/3548729940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d240cbad0 0x7f4d240cd530 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f4d10005fd0 tx=0x7f4d1002a530 comp rx=0 tx=0).stop 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 -- 172.21.15.163:0/3548729940 shutdown_connections 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 --2- 172.21.15.163:0/3548729940 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d240cbad0 0x7f4d240cd530 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 --2- 172.21.15.163:0/3548729940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d24159370 0x7f4d24159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 -- 172.21.15.163:0/3548729940 >> 172.21.15.163:0/3548729940 conn(0x7f4d240b0e30 msgr2=0x7f4d240b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 -- 172.21.15.163:0/3548729940 shutdown_connections 2024-07-26T06:18:56.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.345+0000 7f4d2a400640 1 -- 172.21.15.163:0/3548729940 wait complete. 2024-07-26T06:18:56.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.347+0000 7f4d2a400640 1 Processor -- start 2024-07-26T06:18:56.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.347+0000 7f4d2a400640 1 -- start start 2024-07-26T06:18:56.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d2a400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d240b4420 0x7f4d241f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:56.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d2a400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d24159370 0x7f4d240ac560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:56.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d2a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 0x7f4d24157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:56.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d2a400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4d240cda70 con 0x7f4d24153bb0 2024-07-26T06:18:56.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d2a400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2406f500 con 0x7f4d240b4420 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d2a400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4d240cdbf0 con 0x7f4d24159370 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d29a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 0x7f4d24157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d29a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 0x7f4d24157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35750/0 (socket says 172.21.15.163:35750) 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.348+0000 7f4d29a00640 1 -- 172.21.15.163:0/827215845 learned_addr learned my addr 172.21.15.163:0/827215845 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29000640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d240b4420 0x7f4d241f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29a00640 1 -- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d24159370 msgr2=0x7f4d240ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29a00640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d24159370 0x7f4d240ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29a00640 1 -- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d240b4420 msgr2=0x7f4d241f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29a00640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d240b4420 0x7f4d241f19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29a00640 1 -- 172.21.15.163:0/827215845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d240acaa0 con 0x7f4d24153bb0 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d29a00640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 0x7f4d24157ae0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c780 tx=0x7f4d1c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c017070 con 0x7f4d24153bb0 2024-07-26T06:18:56.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d1c005070 con 0x7f4d24153bb0 2024-07-26T06:18:56.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c0054c0 con 0x7f4d24153bb0 2024-07-26T06:18:56.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.349+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d240acd40 con 0x7f4d24153bb0 2024-07-26T06:18:56.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.350+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d240ad300 con 0x7f4d24153bb0 2024-07-26T06:18:56.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.351+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d241f2b60 con 0x7f4d24153bb0 2024-07-26T06:18:56.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.352+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f4d1c005660 con 0x7f4d24153bb0 2024-07-26T06:18:56.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.353+0000 7f4d20c00640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d04076420 0x7f4d040788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:56.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.353+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4857+0+0 (secure 0 0 0) 0x7f4d1c096b50 con 0x7f4d24153bb0 2024-07-26T06:18:56.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.353+0000 7f4d29000640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d04076420 0x7f4d040788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:56.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.354+0000 7f4d29000640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d04076420 0x7f4d040788e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d180058c0 tx=0x7f4d18005be0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:56.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.357+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d1c0606f0 con 0x7f4d24153bb0 2024-07-26T06:18:56.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:56.547+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4d240b1620 con 0x7f4d24153bb0 2024-07-26T06:18:57.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:18:57.173 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:18:57.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.183+0000 7f4d20c00640 1 -- 172.21.15.163:0/827215845 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) ==== 138+0+0 (secure 0 0 0) 0x7f4d1c05fff0 con 0x7f4d24153bb0 2024-07-26T06:18:57.184 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' created 2024-07-26T06:18:57.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d04076420 msgr2=0x7f4d040788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:57.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d04076420 0x7f4d040788e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d180058c0 tx=0x7f4d18005be0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 msgr2=0x7f4d24157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:57.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 0x7f4d24157ae0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c780 tx=0x7f4d1c00cc40 comp rx=0 tx=0).stop 2024-07-26T06:18:57.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 shutdown_connections 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d04076420 0x7f4d040788e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d24153bb0 0x7f4d24157ae0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d24159370 0x7f4d240ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 --2- 172.21.15.163:0/827215845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d240b4420 0x7f4d241f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 >> 172.21.15.163:0/827215845 conn(0x7f4d240b0e30 msgr2=0x7f4d240b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 shutdown_connections 2024-07-26T06:18:57.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.187+0000 7f4d2a400640 1 -- 172.21.15.163:0/827215845 wait complete. 2024-07-26T06:18:57.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:18:57.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b9a00640 1 -- 172.21.15.163:0/4248780041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb898003a50 msgr2=0x7fb898003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:57.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/4248780041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb898003a50 0x7fb898003e70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400baf0 tx=0x7fb898005830 comp rx=0 tx=0).stop 2024-07-26T06:18:57.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b3400640 1 -- 172.21.15.163:0/4248780041 reap_dead start 2024-07-26T06:18:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b9a00640 1 -- 172.21.15.163:0/4248780041 shutdown_connections 2024-07-26T06:18:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b9a00640 1 -- 172.21.15.163:0/4248780041 >> 172.21.15.163:0/4248780041 conn(0x7fb8b406d420 msgr2=0x7fb8b406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b9a00640 1 -- 172.21.15.163:0/4248780041 shutdown_connections 2024-07-26T06:18:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.458+0000 7fb8b9a00640 1 -- 172.21.15.163:0/4248780041 wait complete. 2024-07-26T06:18:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.460+0000 7fb8b9a00640 1 Processor -- start 2024-07-26T06:18:57.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.460+0000 7fb8b9a00640 1 -- start start 2024-07-26T06:18:57.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b9a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb898003a50 0x7fb8b4158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:57.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b9a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8b4152ff0 0x7fb8b4156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:57.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b9a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 0x7fb8b41573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:57.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b9a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b415eea0 con 0x7fb8b4153450 2024-07-26T06:18:57.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b9a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b415ed20 con 0x7fb898003a50 2024-07-26T06:18:57.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b9a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b415f020 con 0x7fb8b4152ff0 2024-07-26T06:18:57.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 0x7fb8b41573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.461+0000 7fb8b3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 0x7fb8b41573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35774/0 (socket says 172.21.15.163:35774) 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 -- 172.21.15.163:0/3181607952 learned_addr learned my addr 172.21.15.163:0/3181607952 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b2a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb898003a50 0x7fb8b4158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8abe00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8b4152ff0 0x7fb8b4156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 -- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8b4152ff0 msgr2=0x7fb8b4156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8b4152ff0 0x7fb8b4156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 -- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb898003a50 msgr2=0x7fb8b4158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb898003a50 0x7fb8b4158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 -- 172.21.15.163:0/3181607952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b4155d80 con 0x7fb8b4153450 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b2a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb898003a50 0x7fb8b4158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.462+0000 7fb8b3400640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 0x7fb8b41573e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac005fd0 tx=0x7fb8ac009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:57.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.463+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac004b20 con 0x7fb8b4153450 2024-07-26T06:18:57.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.463+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8ac004db0 con 0x7fb8b4153450 2024-07-26T06:18:57.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.463+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00e960 con 0x7fb8b4153450 2024-07-26T06:18:57.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.463+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b4157a10 con 0x7fb8b4153450 2024-07-26T06:18:57.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.463+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b41582b0 con 0x7fb8b4153450 2024-07-26T06:18:57.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.464+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b406c370 con 0x7fb8b4153450 2024-07-26T06:18:57.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.466+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb8ac01a020 con 0x7fb8b4153450 2024-07-26T06:18:57.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.467+0000 7fb8aaa00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb8900763f0 0x7fb8900788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:57.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.467+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5227+0+0 (secure 0 0 0) 0x7fb8ac0981e0 con 0x7fb8b4153450 2024-07-26T06:18:57.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.468+0000 7fb8b2a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb8900763f0 0x7fb8900788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:57.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.468+0000 7fb8b2a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb8900763f0 0x7fb8900788b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4036000 tx=0x7fb8a40046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:57.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.470+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8ac061ae0 con 0x7fb8b4153450 2024-07-26T06:18:57.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:57.677+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb8b415b060 con 0x7fb8b4153450 2024-07-26T06:18:58.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.202+0000 7fb8aaa00640 1 -- 172.21.15.163:0/3181607952 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) ==== 170+0+0 (secure 0 0 0) 0x7fb8ac0613e0 con 0x7fb8b4153450 2024-07-26T06:18:58.204 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' created 2024-07-26T06:18:58.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.207+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb8900763f0 msgr2=0x7fb8900788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:58.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.207+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb8900763f0 0x7fb8900788b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4036000 tx=0x7fb8a40046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.207+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 msgr2=0x7fb8b41573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:58.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.207+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 0x7fb8b41573e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac005fd0 tx=0x7fb8ac009a80 comp rx=0 tx=0).stop 2024-07-26T06:18:58.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.209+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 shutdown_connections 2024-07-26T06:18:58.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.209+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb8900763f0 0x7fb8900788b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.209+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8b4153450 0x7fb8b41573e0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.209+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8b4152ff0 0x7fb8b4156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.209+0000 7fb8b9a00640 1 --2- 172.21.15.163:0/3181607952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb898003a50 0x7fb8b4158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.209+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 >> 172.21.15.163:0/3181607952 conn(0x7fb8b406d420 msgr2=0x7fb8b40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:58.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.210+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 shutdown_connections 2024-07-26T06:18:58.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.211+0000 7fb8b9a00640 1 -- 172.21.15.163:0/3181607952 wait complete. 2024-07-26T06:18:58.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:18:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 -- 172.21.15.163:0/886575910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d80cff20 msgr2=0x7ff8d813eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 --2- 172.21.15.163:0/886575910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d80cff20 0x7ff8d813eba0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0005fd0 tx=0x7ff8d002a8b0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 -- 172.21.15.163:0/886575910 shutdown_connections 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 --2- 172.21.15.163:0/886575910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d80cff20 0x7ff8d813eba0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 --2- 172.21.15.163:0/886575910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8d8071270 0x7ff8d8141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 -- 172.21.15.163:0/886575910 >> 172.21.15.163:0/886575910 conn(0x7ff8d806cab0 msgr2=0x7ff8d80b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 -- 172.21.15.163:0/886575910 shutdown_connections 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.429+0000 7ff8de600640 1 -- 172.21.15.163:0/886575910 wait complete. 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.431+0000 7ff8de600640 1 Processor -- start 2024-07-26T06:18:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.431+0000 7ff8de600640 1 -- start start 2024-07-26T06:18:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8de600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8d8071270 0x7ff8d815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8de600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8d80cff20 0x7ff8d813d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8de600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 0x7ff8d80b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8de600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d813f710 con 0x7ff8d815b1e0 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8de600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d813f590 con 0x7ff8d80cff20 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8de600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d813f890 con 0x7ff8d8071270 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 0x7ff8d80b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 0x7ff8d80b59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35790/0 (socket says 172.21.15.163:35790) 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.432+0000 7ff8d7e00640 1 -- 172.21.15.163:0/1796729844 learned_addr learned my addr 172.21.15.163:0/1796729844 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8cfe00640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8d80cff20 0x7ff8d813d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7400640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8d8071270 0x7ff8d815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7e00640 1 -- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8d8071270 msgr2=0x7ff8d815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7e00640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8d8071270 0x7ff8d815d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7e00640 1 -- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8d80cff20 msgr2=0x7ff8d813d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:58.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7e00640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8d80cff20 0x7ff8d813d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:58.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7e00640 1 -- 172.21.15.163:0/1796729844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d80a9570 con 0x7ff8d815b1e0 2024-07-26T06:18:58.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8d7e00640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 0x7ff8d80b59a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009cd0 tx=0x7ff8d0031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0040070 con 0x7ff8d815b1e0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8d0004040 con 0x7ff8d815b1e0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0004470 con 0x7ff8d815b1e0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.433+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d81f5810 con 0x7ff8d815b1e0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.434+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d815dce0 con 0x7ff8d815b1e0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.435+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d813fdb0 con 0x7ff8d815b1e0 2024-07-26T06:18:58.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.436+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7ff8d0039040 con 0x7ff8d815b1e0 2024-07-26T06:18:58.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.437+0000 7ff8cf400640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff8b40763f0 0x7ff8b40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:58.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.437+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5609+0+0 (secure 0 0 0) 0x7ff8d00b8f50 con 0x7ff8d815b1e0 2024-07-26T06:18:58.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.437+0000 7ff8d7400640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff8b40763f0 0x7ff8b40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:58.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.438+0000 7ff8d7400640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff8b40763f0 0x7ff8b40788b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8000f60 tx=0x7ff8c8009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:58.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.442+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff8d81f5810 con 0x7ff8d815b1e0 2024-07-26T06:18:58.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:58.688+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff8d80b0f20 con 0x7ff8d815b1e0 2024-07-26T06:18:59.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.206+0000 7ff8cf400640 1 -- 172.21.15.163:0/1796729844 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v66) ==== 174+0+0 (secure 0 0 0) 0x7ff8d0082780 con 0x7ff8d815b1e0 2024-07-26T06:18:59.208 INFO:teuthology.orchestra.run.smithi163.stderr:new fs with metadata pool 16 and data pool 17 2024-07-26T06:18:59.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.211+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff8b40763f0 msgr2=0x7ff8b40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff8b40763f0 0x7ff8b40788b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8000f60 tx=0x7ff8c8009040 comp rx=0 tx=0).stop 2024-07-26T06:18:59.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 msgr2=0x7ff8d80b59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 0x7ff8d80b59a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009cd0 tx=0x7ff8d0031760 comp rx=0 tx=0).stop 2024-07-26T06:18:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 shutdown_connections 2024-07-26T06:18:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff8b40763f0 0x7ff8b40788b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8d815b1e0 0x7ff8d80b59a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8d80cff20 0x7ff8d813d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 --2- 172.21.15.163:0/1796729844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8d8071270 0x7ff8d815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.212+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 >> 172.21.15.163:0/1796729844 conn(0x7ff8d806cab0 msgr2=0x7ff8d81597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:59.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.213+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 shutdown_connections 2024-07-26T06:18:59.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.213+0000 7ff8de600640 1 -- 172.21.15.163:0/1796729844 wait complete. 2024-07-26T06:18:59.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:59.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.425+0000 7f715f200640 1 -- 172.21.15.163:0/4287101016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7144003a50 msgr2=0x7f7144003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.425+0000 7f715f200640 1 --2- 172.21.15.163:0/4287101016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7144003a50 0x7f7144003e70 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f715000bb20 tx=0x7f7144005830 comp rx=0 tx=0).stop 2024-07-26T06:18:59.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.425+0000 7f715e800640 1 -- 172.21.15.163:0/4287101016 reap_dead start 2024-07-26T06:18:59.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.425+0000 7f715f200640 1 -- 172.21.15.163:0/4287101016 shutdown_connections 2024-07-26T06:18:59.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.425+0000 7f715f200640 1 -- 172.21.15.163:0/4287101016 >> 172.21.15.163:0/4287101016 conn(0x7f71600b0e30 msgr2=0x7f71600b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:59.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.425+0000 7f715f200640 1 -- 172.21.15.163:0/4287101016 shutdown_connections 2024-07-26T06:18:59.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.426+0000 7f715f200640 1 -- 172.21.15.163:0/4287101016 wait complete. 2024-07-26T06:18:59.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.427+0000 7f715f200640 1 Processor -- start 2024-07-26T06:18:59.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.427+0000 7f715f200640 1 -- start start 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.428+0000 7f715f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7144003a50 0x7f716014ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.428+0000 7f715f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71600aacc0 0x7f71600aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.428+0000 7f715f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 0x7f71600aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.428+0000 7f715f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f71600b3c50 con 0x7f71600a9b90 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.428+0000 7f715f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f71600cdfe0 con 0x7f71600aacc0 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.428+0000 7f715f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f71600b3dd0 con 0x7f7144003a50 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 0x7f71600aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 0x7f71600aa620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35806/0 (socket says 172.21.15.163:35806) 2024-07-26T06:18:59.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 -- 172.21.15.163:0/3076252557 learned_addr learned my addr 172.21.15.163:0/3076252557 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715de00640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7144003a50 0x7f716014ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715d400640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71600aacc0 0x7f71600aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 -- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7144003a50 msgr2=0x7f716014ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7144003a50 0x7f716014ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 -- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71600aacc0 msgr2=0x7f71600aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71600aacc0 0x7f71600aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 -- 172.21.15.163:0/3076252557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716015a660 con 0x7f71600a9b90 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715d400640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71600aacc0 0x7f71600aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.429+0000 7f715e800640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 0x7f71600aa620 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f71600b1e00 tx=0x7f715800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.430+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158004b20 con 0x7f71600a9b90 2024-07-26T06:18:59.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.430+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7158004dd0 con 0x7f71600a9b90 2024-07-26T06:18:59.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.430+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715800cc30 con 0x7f71600a9b90 2024-07-26T06:18:59.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.430+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716015a940 con 0x7f71600a9b90 2024-07-26T06:18:59.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.430+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716015af00 con 0x7f71600a9b90 2024-07-26T06:18:59.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.431+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716015b100 con 0x7f71600a9b90 2024-07-26T06:18:59.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.432+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f715801b020 con 0x7f71600a9b90 2024-07-26T06:18:59.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.433+0000 7f7156000640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7140076420 0x7f71400788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.433+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f7158097630 con 0x7f71600a9b90 2024-07-26T06:18:59.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.433+0000 7f715de00640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7140076420 0x7f71400788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.434+0000 7f715de00640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7140076420 0x7f71400788e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7150036000 tx=0x7f71500046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:59.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.438+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f716015b100 con 0x7f71600a9b90 2024-07-26T06:18:59.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.629+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7160071870 con 0x7f71600a9b90 2024-07-26T06:18:59.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.630+0000 7f7156000640 1 -- 172.21.15.163:0/3076252557 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16653 (secure 0 0 0) 0x7f7158060cd0 con 0x7f71600a9b90 2024-07-26T06:18:59.631 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:18:59.631 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":98,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:58.690277+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:56.563057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:57.693372+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:18:59.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.632+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7140076420 msgr2=0x7f71400788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.632+0000 7f715f200640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7140076420 0x7f71400788e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7150036000 tx=0x7f71500046c0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.632+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 msgr2=0x7f71600aa620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.632+0000 7f715f200640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 0x7f71600aa620 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f71600b1e00 tx=0x7f715800ed20 comp rx=0 tx=0).stop 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 shutdown_connections 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7140076420 0x7f71400788e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71600a9b90 0x7f71600aa620 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71600aacc0 0x7f71600aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 --2- 172.21.15.163:0/3076252557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7144003a50 0x7f716014ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 >> 172.21.15.163:0/3076252557 conn(0x7f71600b0e30 msgr2=0x7f71600b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 shutdown_connections 2024-07-26T06:18:59.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.633+0000 7f715f200640 1 -- 172.21.15.163:0/3076252557 wait complete. 2024-07-26T06:18:59.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 -- 172.21.15.163:0/2176960077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 msgr2=0x7f1ce80b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2176960077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80b3710 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0005fd0 tx=0x7f1ce002a8a0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 -- 172.21.15.163:0/2176960077 shutdown_connections 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2176960077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80b3710 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2176960077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ce81556f0 0x7f1ce80cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 -- 172.21.15.163:0/2176960077 >> 172.21.15.163:0/2176960077 conn(0x7f1ce80b0e30 msgr2=0x7f1ce80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:18:59.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 -- 172.21.15.163:0/2176960077 shutdown_connections 2024-07-26T06:18:59.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.851+0000 7f1ceca00640 1 -- 172.21.15.163:0/2176960077 wait complete. 2024-07-26T06:18:59.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.852+0000 7f1ceca00640 1 Processor -- start 2024-07-26T06:18:59.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.853+0000 7f1ceca00640 1 -- start start 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.853+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ce80ce2d0 0x7f1ce8159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ce81556f0 0x7f1ce80ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ceca00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce80b4020 con 0x7f1ce8071c40 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ceca00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce80b3ea0 con 0x7f1ce81556f0 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ceca00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce80b41a0 con 0x7f1ce80ce2d0 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80aad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35824/0 (socket says 172.21.15.163:35824) 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 -- 172.21.15.163:0/2309942507 learned_addr learned my addr 172.21.15.163:0/2309942507 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1cdea00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ce80ce2d0 0x7f1ce8159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7e00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ce81556f0 0x7f1ce80ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 -- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ce80ce2d0 msgr2=0x7f1ce8159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ce80ce2d0 0x7f1ce8159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 -- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ce81556f0 msgr2=0x7f1ce80ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ce81556f0 0x7f1ce80ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:18:59.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.854+0000 7f1ce7400640 1 -- 172.21.15.163:0/2309942507 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce80abe20 con 0x7f1ce8071c40 2024-07-26T06:18:59.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.855+0000 7f1ce7400640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80aad70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800c970 tx=0x7f1cd800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:59.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.855+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8007bb0 con 0x7f1ce8071c40 2024-07-26T06:18:59.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.855+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1cd8007e40 con 0x7f1ce8071c40 2024-07-26T06:18:59.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.855+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd800f690 con 0x7f1ce8071c40 2024-07-26T06:18:59.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.855+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce8158ca0 con 0x7f1ce8071c40 2024-07-26T06:18:59.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.855+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce8159260 con 0x7f1ce8071c40 2024-07-26T06:18:59.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.856+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce80b46c0 con 0x7f1ce8071c40 2024-07-26T06:18:59.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.857+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f1cd800f830 con 0x7f1ce8071c40 2024-07-26T06:18:59.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.858+0000 7f1cdfe00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1cc4076420 0x7f1cc40788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:18:59.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.859+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f1cd8098130 con 0x7f1ce8071c40 2024-07-26T06:18:59.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.859+0000 7f1cdea00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1cc4076420 0x7f1cc40788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:18:59.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.859+0000 7f1cdea00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1cc4076420 0x7f1cc40788e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80cdd60 tx=0x7f1cd0009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:18:59.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:18:59.862+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cd8014030 con 0x7f1ce8071c40 2024-07-26T06:19:00.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.070+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ce80b34f0 con 0x7f1ce8071c40 2024-07-26T06:19:00.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.073+0000 7f1cdfe00640 1 -- 172.21.15.163:0/2309942507 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16653 (secure 0 0 0) 0x7f1cd8061860 con 0x7f1ce8071c40 2024-07-26T06:19:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:00.075 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":98,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:18:58.690277+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:56.563057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:57.693372+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:19:00.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1cc4076420 msgr2=0x7f1cc40788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:00.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1cc4076420 0x7f1cc40788e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80cdd60 tx=0x7f1cd0009040 comp rx=0 tx=0).stop 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 msgr2=0x7f1ce80aad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80aad70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800c970 tx=0x7f1cd800ce30 comp rx=0 tx=0).stop 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 shutdown_connections 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1cc4076420 0x7f1cc40788e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ce81556f0 0x7f1ce80ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ce80ce2d0 0x7f1ce8159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 --2- 172.21.15.163:0/2309942507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1ce8071c40 0x7f1ce80aad70 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.076+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 >> 172.21.15.163:0/2309942507 conn(0x7f1ce80b0e30 msgr2=0x7f1ce8151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.077+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 shutdown_connections 2024-07-26T06:19:00.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.077+0000 7f1ceca00640 1 -- 172.21.15.163:0/2309942507 wait complete. 2024-07-26T06:19:00.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.293+0000 7fe777e00640 1 -- 172.21.15.163:0/2682278810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 msgr2=0x7fe7780cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.293+0000 7fe777e00640 1 --2- 172.21.15.163:0/2682278810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7780cda70 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe77806a0c0 tx=0x7fe76402a560 comp rx=0 tx=0).stop 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 -- 172.21.15.163:0/2682278810 shutdown_connections 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 --2- 172.21.15.163:0/2682278810 >> v2:172.21.15.164:3300/0 conn(0x7fe7780cd000 0x7fe77815ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 --2- 172.21.15.163:0/2682278810 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7780cda70 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 --2- 172.21.15.163:0/2682278810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7780cbad0 0x7fe7780cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 -- 172.21.15.163:0/2682278810 >> 172.21.15.163:0/2682278810 conn(0x7fe7780b0e30 msgr2=0x7fe7780b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 -- 172.21.15.163:0/2682278810 shutdown_connections 2024-07-26T06:19:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.294+0000 7fe777e00640 1 -- 172.21.15.163:0/2682278810 wait complete. 2024-07-26T06:19:00.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.295+0000 7fe777e00640 1 Processor -- start 2024-07-26T06:19:00.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.295+0000 7fe777e00640 1 -- start start 2024-07-26T06:19:00.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.296+0000 7fe777e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7781499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:00.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.296+0000 7fe777e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7780cd000 0x7fe7780adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:00.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.296+0000 7fe777e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7780728b0 0x7fe778072d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:00.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.296+0000 7fe777e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe778159700 con 0x7fe7780b4420 2024-07-26T06:19:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.296+0000 7fe777e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe778159580 con 0x7fe7780728b0 2024-07-26T06:19:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.296+0000 7fe777e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe778159880 con 0x7fe7780cd000 2024-07-26T06:19:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7781499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7781499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35848/0 (socket says 172.21.15.163:35848) 2024-07-26T06:19:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 -- 172.21.15.163:0/3059174274 learned_addr learned my addr 172.21.15.163:0/3059174274 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 -- 172.21.15.163:0/3059174274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7780cd000 msgr2=0x7fe7780adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7780cd000 0x7fe7780adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 -- 172.21.15.163:0/3059174274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7780728b0 msgr2=0x7fe778072d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7780728b0 0x7fe778072d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 -- 172.21.15.163:0/3059174274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77814a000 con 0x7fe7780b4420 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe776a00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7781499d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe76800c940 tx=0x7fe76800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe768007bb0 con 0x7fe7780b4420 2024-07-26T06:19:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.297+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe768005020 con 0x7fe7780b4420 2024-07-26T06:19:00.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.298+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe778151ce0 con 0x7fe7780b4420 2024-07-26T06:19:00.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.298+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe768007e60 con 0x7fe7780b4420 2024-07-26T06:19:00.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.298+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7781522a0 con 0x7fe7780b4420 2024-07-26T06:19:00.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.299+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77815ce50 con 0x7fe7780b4420 2024-07-26T06:19:00.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.300+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fe768005470 con 0x7fe7780b4420 2024-07-26T06:19:00.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.301+0000 7fe76f400640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe758076420 0x7fe7580788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:00.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.301+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fe768097e40 con 0x7fe7780b4420 2024-07-26T06:19:00.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.301+0000 7fe776000640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe758076420 0x7fe7580788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:00.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.302+0000 7fe776000640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe758076420 0x7fe7580788e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe764033000 tx=0x7fe7640046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:00.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.305+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe768014030 con 0x7fe7780b4420 2024-07-26T06:19:00.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:00.551+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe77815ba90 con 0x7fe7780b4420 2024-07-26T06:19:00.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-07-26T06:19:01.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.250+0000 7fe76f400640 1 -- 172.21.15.163:0/3059174274 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) ==== 118+0+0 (secure 0 0 0) 0x7fe768061570 con 0x7fe7780b4420 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.253+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe758076420 msgr2=0x7fe7580788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.253+0000 7fe777e00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe758076420 0x7fe7580788e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe764033000 tx=0x7fe7640046c0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.253+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 msgr2=0x7fe7781499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.253+0000 7fe777e00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7781499d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe76800c940 tx=0x7fe76800ce00 comp rx=0 tx=0).stop 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 shutdown_connections 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe758076420 0x7fe7580788e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7780728b0 0x7fe778072d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7780cd000 0x7fe7780adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 --2- 172.21.15.163:0/3059174274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7780b4420 0x7fe7781499d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 >> 172.21.15.163:0/3059174274 conn(0x7fe7780b0e30 msgr2=0x7fe77815a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 shutdown_connections 2024-07-26T06:19:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.254+0000 7fe777e00640 1 -- 172.21.15.163:0/3059174274 wait complete. 2024-07-26T06:19:01.276 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-07-26T06:19:01.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 -- 172.21.15.163:0/2524358527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3730004de0 msgr2=0x7f37300071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 --2- 172.21.15.163:0/2524358527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3730004de0 0x7f37300071d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f373c0016c0 tx=0x7f3730003810 comp rx=0 tx=0).stop 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 -- 172.21.15.163:0/2524358527 shutdown_connections 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 --2- 172.21.15.163:0/2524358527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3730004de0 0x7f37300071d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 -- 172.21.15.163:0/2524358527 >> 172.21.15.163:0/2524358527 conn(0x7f374c06d420 msgr2=0x7f374c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 -- 172.21.15.163:0/2524358527 shutdown_connections 2024-07-26T06:19:01.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.489+0000 7f3753200640 1 -- 172.21.15.163:0/2524358527 wait complete. 2024-07-26T06:19:01.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.491+0000 7f3753200640 1 Processor -- start 2024-07-26T06:19:01.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.491+0000 7f3753200640 1 -- start start 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f3753200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3730004de0 0x7f374c14f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f3753200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 0x7f374c0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f3753200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c0b22d0 0x7f374c0b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f3753200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f374c06c0c0 con 0x7f374c0b1160 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f3753200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f374c06bf40 con 0x7f374c0b22d0 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f3753200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f374c06c240 con 0x7f3730004de0 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 0x7f374c0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 0x7f374c0b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35866/0 (socket says 172.21.15.163:35866) 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 -- 172.21.15.163:0/3427346435 learned_addr learned my addr 172.21.15.163:0/3427346435 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 -- 172.21.15.163:0/3427346435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3730004de0 msgr2=0x7f374c14f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:01.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3730004de0 0x7f374c14f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 -- 172.21.15.163:0/3427346435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c0b22d0 msgr2=0x7f374c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c0b22d0 0x7f374c0b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.492+0000 7f374b400640 1 -- 172.21.15.163:0/3427346435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c0af410 con 0x7f374c0b1160 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.493+0000 7f374b400640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 0x7f374c0b15c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f373800e710 tx=0x7f373800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.493+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373800cab0 con 0x7f374c0b1160 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.493+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f373800cc50 con 0x7f374c0b1160 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.493+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738005020 con 0x7f374c0b1160 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.493+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374c0af6f0 con 0x7f374c0b1160 2024-07-26T06:19:01.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.493+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c0afcb0 con 0x7f374c0b1160 2024-07-26T06:19:01.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.494+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f374c0b42b0 con 0x7f374c0b1160 2024-07-26T06:19:01.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.495+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f37380053d0 con 0x7f374c0b1160 2024-07-26T06:19:01.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.496+0000 7f3743e00640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f372c076420 0x7f372c0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:01.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.496+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7f3738098420 con 0x7f374c0b1160 2024-07-26T06:19:01.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.497+0000 7f374be00640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f372c076420 0x7f372c0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:01.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.497+0000 7f374be00640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f372c076420 0x7f372c0788e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f373c032000 tx=0x7f373c001120 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:01.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.501+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3738061a30 con 0x7f374c0b1160 2024-07-26T06:19:01.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:01.752+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f374c1e6280 con 0x7f374c0b1160 2024-07-26T06:19:02.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.254+0000 7f3743e00640 1 -- 172.21.15.163:0/3427346435 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) ==== 122+0+0 (secure 0 0 0) 0x7f3738061330 con 0x7f374c0b1160 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f372c076420 msgr2=0x7f372c0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f372c076420 0x7f372c0788e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f373c032000 tx=0x7f373c001120 comp rx=0 tx=0).stop 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 msgr2=0x7f374c0b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 0x7f374c0b15c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f373800e710 tx=0x7f373800ebd0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 shutdown_connections 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f372c076420 0x7f372c0788e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c0b22d0 0x7f374c0b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f374c0b1160 0x7f374c0b15c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 --2- 172.21.15.163:0/3427346435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3730004de0 0x7f374c14f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 >> 172.21.15.163:0/3427346435 conn(0x7f374c06d420 msgr2=0x7f374c1e9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:02.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 shutdown_connections 2024-07-26T06:19:02.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.258+0000 7f3753200640 1 -- 172.21.15.163:0/3427346435 wait complete. 2024-07-26T06:19:02.280 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-07-26T06:19:02.280 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:19:02.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 -- 172.21.15.163:0/245773859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 msgr2=0x7f574c004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 --2- 172.21.15.163:0/245773859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f574c004f80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f57580016c0 tx=0x7f574c006940 comp rx=0 tx=0).stop 2024-07-26T06:19:02.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 -- 172.21.15.163:0/245773859 shutdown_connections 2024-07-26T06:19:02.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 --2- 172.21.15.163:0/245773859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f574c004f80 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 -- 172.21.15.163:0/245773859 >> 172.21.15.163:0/245773859 conn(0x7f57680b0e30 msgr2=0x7f57680b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 -- 172.21.15.163:0/245773859 shutdown_connections 2024-07-26T06:19:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.477+0000 7f576ca00640 1 -- 172.21.15.163:0/245773859 wait complete. 2024-07-26T06:19:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.479+0000 7f576ca00640 1 Processor -- start 2024-07-26T06:19:02.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.479+0000 7f576ca00640 1 -- start start 2024-07-26T06:19:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f576ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f5768143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f576ca00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5768153bf0 0x7f5768154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f576ca00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57681525e0 0x7f5768152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f576ca00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57680cd530 con 0x7f574c004b60 2024-07-26T06:19:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f576ca00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f576806f500 con 0x7f5768153bf0 2024-07-26T06:19:02.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f576ca00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f57680cd6b0 con 0x7f57681525e0 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f5768143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f5768143900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35876/0 (socket says 172.21.15.163:35876) 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 -- 172.21.15.163:0/2874057164 learned_addr learned my addr 172.21.15.163:0/2874057164 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f575de00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5768153bf0 0x7f5768154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5767200640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57681525e0 0x7f5768152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 -- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57681525e0 msgr2=0x7f5768152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57681525e0 0x7f5768152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 -- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5768153bf0 msgr2=0x7f5768154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5768153bf0 0x7f5768154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f5766800640 1 -- 172.21.15.163:0/2874057164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5768155bd0 con 0x7f574c004b60 2024-07-26T06:19:02.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.480+0000 7f575de00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5768153bf0 0x7f5768154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.481+0000 7f5766800640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f5768143900 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f575802a910 tx=0x7f57580063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.481+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758003a00 con 0x7f574c004b60 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.481+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5758039070 con 0x7f574c004b60 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.481+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5768155eb0 con 0x7f574c004b60 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.481+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758033420 con 0x7f574c004b60 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.481+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5768156470 con 0x7f574c004b60 2024-07-26T06:19:02.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.482+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5768159370 con 0x7f574c004b60 2024-07-26T06:19:02.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.483+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f5758044020 con 0x7f574c004b60 2024-07-26T06:19:02.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.484+0000 7f575f400640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5744076420 0x7f57440788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.485+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7f57580b6fe0 con 0x7f574c004b60 2024-07-26T06:19:02.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.485+0000 7f575de00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5744076420 0x7f57440788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.485+0000 7f575de00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5744076420 0x7f57440788e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f576815cb90 tx=0x7f5750005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:02.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.488+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5758080830 con 0x7f574c004b60 2024-07-26T06:19:02.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.731+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57680b35c0 con 0x7f574c004b60 2024-07-26T06:19:02.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.732+0000 7f575f400640 1 -- 172.21.15.163:0/2874057164 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5234 (secure 0 0 0) 0x7f5758080130 con 0x7f574c004b60 2024-07-26T06:19:02.733 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:02.733 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":70,"btime":"2024-07-26T06:19:02:251060+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:58.690202+0000","modified":"2024-07-26T06:19:01.753809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6834/1667488712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1667488712},{"type":"v1","addr":"172.21.15.163:6834","nonce":1667488712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":8}]} 2024-07-26T06:19:02.734 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 70 2024-07-26T06:19:02.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5744076420 msgr2=0x7f57440788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5744076420 0x7f57440788e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f576815cb90 tx=0x7f5750005f90 comp rx=0 tx=0).stop 2024-07-26T06:19:02.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 msgr2=0x7f5768143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f5768143900 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f575802a910 tx=0x7f57580063a0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 shutdown_connections 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5744076420 0x7f57440788e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57681525e0 0x7f5768152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5768153bf0 0x7f5768154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 --2- 172.21.15.163:0/2874057164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574c004b60 0x7f5768143900 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 >> 172.21.15.163:0/2874057164 conn(0x7f57680b0e30 msgr2=0x7f5768159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.735+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 shutdown_connections 2024-07-26T06:19:02.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.736+0000 7f576ca00640 1 -- 172.21.15.163:0/2874057164 wait complete. 2024-07-26T06:19:02.755 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.957+0000 7fb260c00640 1 -- 172.21.15.163:0/1829507299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c071a70 msgr2=0x7fb25c0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.957+0000 7fb260c00640 1 --2- 172.21.15.163:0/1829507299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c071a70 0x7fb25c0b3710 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb254005fd0 tx=0x7fb25402aa00 comp rx=0 tx=0).stop 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.958+0000 7fb260c00640 1 -- 172.21.15.163:0/1829507299 shutdown_connections 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.958+0000 7fb260c00640 1 --2- 172.21.15.163:0/1829507299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c071a70 0x7fb25c0b3710 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.958+0000 7fb260c00640 1 --2- 172.21.15.163:0/1829507299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25c0ce4a0 0x7fb25c071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.958+0000 7fb260c00640 1 -- 172.21.15.163:0/1829507299 >> 172.21.15.163:0/1829507299 conn(0x7fb25c0b0e30 msgr2=0x7fb25c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.958+0000 7fb260c00640 1 -- 172.21.15.163:0/1829507299 shutdown_connections 2024-07-26T06:19:02.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.958+0000 7fb260c00640 1 -- 172.21.15.163:0/1829507299 wait complete. 2024-07-26T06:19:02.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.959+0000 7fb260c00640 1 Processor -- start 2024-07-26T06:19:02.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.960+0000 7fb260c00640 1 -- start start 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.960+0000 7fb260c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb25c071a70 0x7fb25c152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.960+0000 7fb260c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25c0ce4a0 0x7fb25c148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.960+0000 7fb260c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 0x7fb25c1f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb260c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c0b4020 con 0x7fb25c1556f0 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb260c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c0b3ea0 con 0x7fb25c071a70 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb260c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb25c0b41a0 con 0x7fb25c0ce4a0 2024-07-26T06:19:02.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 0x7fb25c1f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 0x7fb25c1f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35894/0 (socket says 172.21.15.163:35894) 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 -- 172.21.15.163:0/2265355033 learned_addr learned my addr 172.21.15.163:0/2265355033 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25b400640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb25c071a70 0x7fb25c152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25aa00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25c0ce4a0 0x7fb25c148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 -- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25c0ce4a0 msgr2=0x7fb25c148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25c0ce4a0 0x7fb25c148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 -- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb25c071a70 msgr2=0x7fb25c152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb25c071a70 0x7fb25c152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25be00640 1 -- 172.21.15.163:0/2265355033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c14df20 con 0x7fb25c1556f0 2024-07-26T06:19:02.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.961+0000 7fb25b400640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb25c071a70 0x7fb25c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:02.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.962+0000 7fb25be00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 0x7fb25c1f7a40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb254006010 tx=0x7fb254003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:02.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.962+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254047070 con 0x7fb25c1556f0 2024-07-26T06:19:02.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.962+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb254004250 con 0x7fb25c1556f0 2024-07-26T06:19:02.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.962+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c14e200 con 0x7fb25c1556f0 2024-07-26T06:19:02.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.962+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254033050 con 0x7fb25c1556f0 2024-07-26T06:19:02.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.962+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25c0ae2b0 con 0x7fb25c1556f0 2024-07-26T06:19:02.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.963+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb25c0b4730 con 0x7fb25c1556f0 2024-07-26T06:19:02.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.964+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb25404a070 con 0x7fb25c1556f0 2024-07-26T06:19:02.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.965+0000 7fb253400640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb238076420 0x7fb2380788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:02.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.965+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fb2540b8b80 con 0x7fb25c1556f0 2024-07-26T06:19:02.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.966+0000 7fb25b400640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb238076420 0x7fb2380788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:02.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.966+0000 7fb25b400640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb238076420 0x7fb2380788e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb25c150340 tx=0x7fb24c008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:02.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:02.969+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb2540822b0 con 0x7fb25c1556f0 2024-07-26T06:19:03.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.176+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb25c0b34f0 con 0x7fb25c1556f0 2024-07-26T06:19:03.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.178+0000 7fb253400640 1 -- 172.21.15.163:0/2265355033 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16640 (secure 0 0 0) 0x7fb254081bb0 con 0x7fb25c1556f0 2024-07-26T06:19:03.179 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:03.179 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":99,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:19:00.226106+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:56.563057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:57.693372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb238076420 msgr2=0x7fb2380788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb238076420 0x7fb2380788e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb25c150340 tx=0x7fb24c008040 comp rx=0 tx=0).stop 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 msgr2=0x7fb25c1f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 0x7fb25c1f7a40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb254006010 tx=0x7fb254003f10 comp rx=0 tx=0).stop 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 shutdown_connections 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb238076420 0x7fb2380788e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb25c1556f0 0x7fb25c1f7a40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25c0ce4a0 0x7fb25c148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 --2- 172.21.15.163:0/2265355033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb25c071a70 0x7fb25c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.180+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 >> 172.21.15.163:0/2265355033 conn(0x7fb25c0b0e30 msgr2=0x7fb25c0ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:03.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.181+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 shutdown_connections 2024-07-26T06:19:03.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.181+0000 7fb260c00640 1 -- 172.21.15.163:0/2265355033 wait complete. 2024-07-26T06:19:03.202 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:19:03.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:19:03.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 -- 172.21.15.163:0/2969614547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 msgr2=0x7fbe940b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2969614547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe940b4160 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fbe9406de60 tx=0x7fbe8002a560 comp rx=0 tx=0).stop 2024-07-26T06:19:03.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 -- 172.21.15.163:0/2969614547 shutdown_connections 2024-07-26T06:19:03.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2969614547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe940b46a0 0x7fbe94072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2969614547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe940b4160 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2969614547 >> v2:172.21.15.164:3300/0 conn(0x7fbe9415afa0 0x7fbe9415a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 -- 172.21.15.163:0/2969614547 >> 172.21.15.163:0/2969614547 conn(0x7fbe9406d050 msgr2=0x7fbe9406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:03.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 -- 172.21.15.163:0/2969614547 shutdown_connections 2024-07-26T06:19:03.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.401+0000 7fbe93e00640 1 -- 172.21.15.163:0/2969614547 wait complete. 2024-07-26T06:19:03.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.403+0000 7fbe93e00640 1 Processor -- start 2024-07-26T06:19:03.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.403+0000 7fbe93e00640 1 -- start start 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe9415afa0 0x7fbe94162fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe94142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe94142bd0 0x7fbe941f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9415b380 con 0x7fbe9415dae0 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94072d70 con 0x7fbe9415afa0 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9415b500 con 0x7fbe94142bd0 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe94142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe94142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35906/0 (socket says 172.21.15.163:35906) 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 -- 172.21.15.163:0/2691036257 learned_addr learned my addr 172.21.15.163:0/2691036257 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:03.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92a00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe9415afa0 0x7fbe94162fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe93400640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe94142bd0 0x7fbe941f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 -- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe94142bd0 msgr2=0x7fbe941f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe94142bd0 0x7fbe941f4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 -- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe9415afa0 msgr2=0x7fbe94162fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe9415afa0 0x7fbe94162fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.404+0000 7fbe92000640 1 -- 172.21.15.163:0/2691036257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe94161c50 con 0x7fbe9415dae0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.405+0000 7fbe92000640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe94142690 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009850 tx=0x7fbe80003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.405+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8003f070 con 0x7fbe9415dae0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.405+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe800047a0 con 0x7fbe9415dae0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.405+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe80035030 con 0x7fbe9415dae0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.405+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe94161f30 con 0x7fbe9415dae0 2024-07-26T06:19:03.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.405+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941624f0 con 0x7fbe9415dae0 2024-07-26T06:19:03.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.406+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9415a4a0 con 0x7fbe9415dae0 2024-07-26T06:19:03.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.407+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fbe80042070 con 0x7fbe9415dae0 2024-07-26T06:19:03.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.408+0000 7fbe8b400640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbe740763f0 0x7fbe740788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.408+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7fbe800b8840 con 0x7fbe9415dae0 2024-07-26T06:19:03.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.409+0000 7fbe92a00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbe740763f0 0x7fbe740788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.409+0000 7fbe92a00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbe740763f0 0x7fbe740788b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbe84005fd0 tx=0x7fbe84005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:03.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.413+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe80081ee0 con 0x7fbe9415dae0 2024-07-26T06:19:03.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.623+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe9415aa40 con 0x7fbe740763f0 2024-07-26T06:19:03.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.635+0000 7fbe8b400640 1 -- 172.21.15.163:0/2691036257 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252038 (secure 0 0 0) 0x7fbe9415aa40 con 0x7fbe740763f0 2024-07-26T06:19:03.640 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:03.641 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.641+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbe740763f0 msgr2=0x7fbe740788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.641+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbe740763f0 0x7fbe740788b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbe84005fd0 tx=0x7fbe84005850 comp rx=0 tx=0).stop 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.641+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 msgr2=0x7fbe94142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.641+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe94142690 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009850 tx=0x7fbe80003910 comp rx=0 tx=0).stop 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 shutdown_connections 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbe740763f0 0x7fbe740788b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe94142bd0 0x7fbe941f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe9415dae0 0x7fbe94142690 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 --2- 172.21.15.163:0/2691036257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe9415afa0 0x7fbe94162fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 >> 172.21.15.163:0/2691036257 conn(0x7fbe9406d050 msgr2=0x7fbe9406e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:03.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 shutdown_connections 2024-07-26T06:19:03.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.642+0000 7fbe93e00640 1 -- 172.21.15.163:0/2691036257 wait complete. 2024-07-26T06:19:03.663 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-07-26T06:19:02.474754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260188,"kb_used_data":10696,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714532,"statfs":{"total":773094113280,"available":772827680768,"internally_reserved":0,"allocated":10952704,"data_stored":5413068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.288971"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245385+0000","last_change":"2024-07-26T06:18:59.243519+0000","last_active":"2024-07-26T06:19:00.245385+0000","last_peered":"2024-07-26T06:19:00.245385+0000","last_clean":"2024-07-26T06:19:00.245385+0000","last_became_active":"2024-07-26T06:18:59.235918+0000","last_became_peered":"2024-07-26T06:18:59.235918+0000","last_unstale":"2024-07-26T06:19:00.245385+0000","last_undegraded":"2024-07-26T06:19:00.245385+0000","last_fullsized":"2024-07-26T06:19:00.245385+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:27:54.756486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.246490+0000","last_change":"2024-07-26T06:18:59.240213+0000","last_active":"2024-07-26T06:19:00.246490+0000","last_peered":"2024-07-26T06:19:00.246490+0000","last_clean":"2024-07-26T06:19:00.246490+0000","last_became_active":"2024-07-26T06:18:59.237248+0000","last_became_peered":"2024-07-26T06:18:59.237248+0000","last_unstale":"2024-07-26T06:19:00.246490+0000","last_undegraded":"2024-07-26T06:19:00.246490+0000","last_fullsized":"2024-07-26T06:19:00.246490+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:48.869491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255767+0000","last_change":"2024-07-26T06:18:59.248831+0000","last_active":"2024-07-26T06:18:59.255767+0000","last_peered":"2024-07-26T06:18:59.255767+0000","last_clean":"2024-07-26T06:18:59.255767+0000","last_became_active":"2024-07-26T06:18:59.248572+0000","last_became_peered":"2024-07-26T06:18:59.248572+0000","last_unstale":"2024-07-26T06:18:59.255767+0000","last_undegraded":"2024-07-26T06:18:59.255767+0000","last_fullsized":"2024-07-26T06:18:59.255767+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:33.396453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244106+0000","last_change":"2024-07-26T06:18:59.236444+0000","last_active":"2024-07-26T06:19:00.244106+0000","last_peered":"2024-07-26T06:19:00.244106+0000","last_clean":"2024-07-26T06:19:00.244106+0000","last_became_active":"2024-07-26T06:18:59.234730+0000","last_became_peered":"2024-07-26T06:18:59.234730+0000","last_unstale":"2024-07-26T06:19:00.244106+0000","last_undegraded":"2024-07-26T06:19:00.244106+0000","last_fullsized":"2024-07-26T06:19:00.244106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:02:02.418506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251244+0000","last_change":"2024-07-26T06:18:59.248389+0000","last_active":"2024-07-26T06:18:59.251244+0000","last_peered":"2024-07-26T06:18:59.251244+0000","last_clean":"2024-07-26T06:18:59.251244+0000","last_became_active":"2024-07-26T06:18:59.248072+0000","last_became_peered":"2024-07-26T06:18:59.248072+0000","last_unstale":"2024-07-26T06:18:59.251244+0000","last_undegraded":"2024-07-26T06:18:59.251244+0000","last_fullsized":"2024-07-26T06:18:59.251244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:45:40.679563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251922+0000","last_change":"2024-07-26T06:18:59.243256+0000","last_active":"2024-07-26T06:18:59.251922+0000","last_peered":"2024-07-26T06:18:59.251922+0000","last_clean":"2024-07-26T06:18:59.251922+0000","last_became_active":"2024-07-26T06:18:59.242688+0000","last_became_peered":"2024-07-26T06:18:59.242688+0000","last_unstale":"2024-07-26T06:18:59.251922+0000","last_undegraded":"2024-07-26T06:18:59.251922+0000","last_fullsized":"2024-07-26T06:18:59.251922+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:21:43.549583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.253243+0000","last_change":"2024-07-26T06:18:59.248843+0000","last_active":"2024-07-26T06:18:59.253243+0000","last_peered":"2024-07-26T06:18:59.253243+0000","last_clean":"2024-07-26T06:18:59.253243+0000","last_became_active":"2024-07-26T06:18:59.248300+0000","last_became_peered":"2024-07-26T06:18:59.248300+0000","last_unstale":"2024-07-26T06:18:59.253243+0000","last_undegraded":"2024-07-26T06:18:59.253243+0000","last_fullsized":"2024-07-26T06:18:59.253243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:44:50.902450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.256488+0000","last_change":"2024-07-26T06:18:59.255038+0000","last_active":"2024-07-26T06:18:59.256488+0000","last_peered":"2024-07-26T06:18:59.256488+0000","last_clean":"2024-07-26T06:18:59.256488+0000","last_became_active":"2024-07-26T06:18:59.254167+0000","last_became_peered":"2024-07-26T06:18:59.254167+0000","last_unstale":"2024-07-26T06:18:59.256488+0000","last_undegraded":"2024-07-26T06:18:59.256488+0000","last_fullsized":"2024-07-26T06:18:59.256488+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:29.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245385+0000","last_change":"2024-07-26T06:18:59.247197+0000","last_active":"2024-07-26T06:19:00.245385+0000","last_peered":"2024-07-26T06:19:00.245385+0000","last_clean":"2024-07-26T06:19:00.245385+0000","last_became_active":"2024-07-26T06:18:59.246611+0000","last_became_peered":"2024-07-26T06:18:59.246611+0000","last_unstale":"2024-07-26T06:19:00.245385+0000","last_undegraded":"2024-07-26T06:19:00.245385+0000","last_fullsized":"2024-07-26T06:19:00.245385+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:53:07.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244597+0000","last_change":"2024-07-26T06:18:59.231835+0000","last_active":"2024-07-26T06:19:00.244597+0000","last_peered":"2024-07-26T06:19:00.244597+0000","last_clean":"2024-07-26T06:19:00.244597+0000","last_became_active":"2024-07-26T06:18:59.231553+0000","last_became_peered":"2024-07-26T06:18:59.231553+0000","last_unstale":"2024-07-26T06:19:00.244597+0000","last_undegraded":"2024-07-26T06:19:00.244597+0000","last_fullsized":"2024-07-26T06:19:00.244597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:44.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.246324+0000","last_change":"2024-07-26T06:18:59.239507+0000","last_active":"2024-07-26T06:19:00.246324+0000","last_peered":"2024-07-26T06:19:00.246324+0000","last_clean":"2024-07-26T06:19:00.246324+0000","last_became_active":"2024-07-26T06:18:59.236138+0000","last_became_peered":"2024-07-26T06:18:59.236138+0000","last_unstale":"2024-07-26T06:19:00.246324+0000","last_undegraded":"2024-07-26T06:19:00.246324+0000","last_fullsized":"2024-07-26T06:19:00.246324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:29.396937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244702+0000","last_change":"2024-07-26T06:18:59.231512+0000","last_active":"2024-07-26T06:19:00.244702+0000","last_peered":"2024-07-26T06:19:00.244702+0000","last_clean":"2024-07-26T06:19:00.244702+0000","last_became_active":"2024-07-26T06:18:59.231141+0000","last_became_peered":"2024-07-26T06:18:59.231141+0000","last_unstale":"2024-07-26T06:19:00.244702+0000","last_undegraded":"2024-07-26T06:19:00.244702+0000","last_fullsized":"2024-07-26T06:19:00.244702+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:01:58.911434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.257520+0000","last_change":"2024-07-26T06:18:59.256066+0000","last_active":"2024-07-26T06:18:59.257520+0000","last_peered":"2024-07-26T06:18:59.257520+0000","last_clean":"2024-07-26T06:18:59.257520+0000","last_became_active":"2024-07-26T06:18:59.255754+0000","last_became_peered":"2024-07-26T06:18:59.255754+0000","last_unstale":"2024-07-26T06:18:59.257520+0000","last_undegraded":"2024-07-26T06:18:59.257520+0000","last_fullsized":"2024-07-26T06:18:59.257520+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:55:33.718740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255990+0000","last_change":"2024-07-26T06:18:59.252936+0000","last_active":"2024-07-26T06:18:59.255990+0000","last_peered":"2024-07-26T06:18:59.255990+0000","last_clean":"2024-07-26T06:18:59.255990+0000","last_became_active":"2024-07-26T06:18:59.252571+0000","last_became_peered":"2024-07-26T06:18:59.252571+0000","last_unstale":"2024-07-26T06:18:59.255990+0000","last_undegraded":"2024-07-26T06:18:59.255990+0000","last_fullsized":"2024-07-26T06:18:59.255990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:28:56.774761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252007+0000","last_change":"2024-07-26T06:18:59.248483+0000","last_active":"2024-07-26T06:18:59.252007+0000","last_peered":"2024-07-26T06:18:59.252007+0000","last_clean":"2024-07-26T06:18:59.252007+0000","last_became_active":"2024-07-26T06:18:59.248218+0000","last_became_peered":"2024-07-26T06:18:59.248218+0000","last_unstale":"2024-07-26T06:18:59.252007+0000","last_undegraded":"2024-07-26T06:18:59.252007+0000","last_fullsized":"2024-07-26T06:18:59.252007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:53:28.329133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242626+0000","last_change":"2024-07-26T06:18:59.244881+0000","last_active":"2024-07-26T06:19:00.242626+0000","last_peered":"2024-07-26T06:19:00.242626+0000","last_clean":"2024-07-26T06:19:00.242626+0000","last_became_active":"2024-07-26T06:18:59.244485+0000","last_became_peered":"2024-07-26T06:18:59.244485+0000","last_unstale":"2024-07-26T06:19:00.242626+0000","last_undegraded":"2024-07-26T06:19:00.242626+0000","last_fullsized":"2024-07-26T06:19:00.242626+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:26.825732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.246621+0000","last_change":"2024-07-26T06:18:59.236328+0000","last_active":"2024-07-26T06:19:00.246621+0000","last_peered":"2024-07-26T06:19:00.246621+0000","last_clean":"2024-07-26T06:19:00.246621+0000","last_became_active":"2024-07-26T06:18:59.235877+0000","last_became_peered":"2024-07-26T06:18:59.235877+0000","last_unstale":"2024-07-26T06:19:00.246621+0000","last_undegraded":"2024-07-26T06:19:00.246621+0000","last_fullsized":"2024-07-26T06:19:00.246621+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:48:38.222129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.247696+0000","last_change":"2024-07-26T06:18:59.240906+0000","last_active":"2024-07-26T06:18:59.247696+0000","last_peered":"2024-07-26T06:18:59.247696+0000","last_clean":"2024-07-26T06:18:59.247696+0000","last_became_active":"2024-07-26T06:18:59.240643+0000","last_became_peered":"2024-07-26T06:18:59.240643+0000","last_unstale":"2024-07-26T06:18:59.247696+0000","last_undegraded":"2024-07-26T06:18:59.247696+0000","last_fullsized":"2024-07-26T06:18:59.247696+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:02.107634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.257080+0000","last_change":"2024-07-26T06:18:59.251750+0000","last_active":"2024-07-26T06:18:59.257080+0000","last_peered":"2024-07-26T06:18:59.257080+0000","last_clean":"2024-07-26T06:18:59.257080+0000","last_became_active":"2024-07-26T06:18:59.250623+0000","last_became_peered":"2024-07-26T06:18:59.250623+0000","last_unstale":"2024-07-26T06:18:59.257080+0000","last_undegraded":"2024-07-26T06:18:59.257080+0000","last_fullsized":"2024-07-26T06:18:59.257080+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:13.948724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245833+0000","last_change":"2024-07-26T06:18:59.246414+0000","last_active":"2024-07-26T06:19:00.245833+0000","last_peered":"2024-07-26T06:19:00.245833+0000","last_clean":"2024-07-26T06:19:00.245833+0000","last_became_active":"2024-07-26T06:18:59.244670+0000","last_became_peered":"2024-07-26T06:18:59.244670+0000","last_unstale":"2024-07-26T06:19:00.245833+0000","last_undegraded":"2024-07-26T06:19:00.245833+0000","last_fullsized":"2024-07-26T06:19:00.245833+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:04:53.973898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245840+0000","last_change":"2024-07-26T06:18:59.245794+0000","last_active":"2024-07-26T06:19:00.245840+0000","last_peered":"2024-07-26T06:19:00.245840+0000","last_clean":"2024-07-26T06:19:00.245840+0000","last_became_active":"2024-07-26T06:18:59.245303+0000","last_became_peered":"2024-07-26T06:18:59.245303+0000","last_unstale":"2024-07-26T06:19:00.245840+0000","last_undegraded":"2024-07-26T06:19:00.245840+0000","last_fullsized":"2024-07-26T06:19:00.245840+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:18.725897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244487+0000","last_change":"2024-07-26T06:18:59.239312+0000","last_active":"2024-07-26T06:19:00.244487+0000","last_peered":"2024-07-26T06:19:00.244487+0000","last_clean":"2024-07-26T06:19:00.244487+0000","last_became_active":"2024-07-26T06:18:59.238852+0000","last_became_peered":"2024-07-26T06:18:59.238852+0000","last_unstale":"2024-07-26T06:19:00.244487+0000","last_undegraded":"2024-07-26T06:19:00.244487+0000","last_fullsized":"2024-07-26T06:19:00.244487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:36:25.099789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244837+0000","last_change":"2024-07-26T06:18:59.251462+0000","last_active":"2024-07-26T06:19:00.244837+0000","last_peered":"2024-07-26T06:19:00.244837+0000","last_clean":"2024-07-26T06:19:00.244837+0000","last_became_active":"2024-07-26T06:18:59.246782+0000","last_became_peered":"2024-07-26T06:18:59.246782+0000","last_unstale":"2024-07-26T06:19:00.244837+0000","last_undegraded":"2024-07-26T06:19:00.244837+0000","last_fullsized":"2024-07-26T06:19:00.244837+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:53.912448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244315+0000","last_change":"2024-07-26T06:18:59.247489+0000","last_active":"2024-07-26T06:19:00.244315+0000","last_peered":"2024-07-26T06:19:00.244315+0000","last_clean":"2024-07-26T06:19:00.244315+0000","last_became_active":"2024-07-26T06:18:59.247009+0000","last_became_peered":"2024-07-26T06:18:59.247009+0000","last_unstale":"2024-07-26T06:19:00.244315+0000","last_undegraded":"2024-07-26T06:19:00.244315+0000","last_fullsized":"2024-07-26T06:19:00.244315+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:40:33.922534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.247002+0000","last_change":"2024-07-26T06:18:59.232772+0000","last_active":"2024-07-26T06:19:00.247002+0000","last_peered":"2024-07-26T06:19:00.247002+0000","last_clean":"2024-07-26T06:19:00.247002+0000","last_became_active":"2024-07-26T06:18:59.232446+0000","last_became_peered":"2024-07-26T06:18:59.232446+0000","last_unstale":"2024-07-26T06:19:00.247002+0000","last_undegraded":"2024-07-26T06:19:00.247002+0000","last_fullsized":"2024-07-26T06:19:00.247002+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:19:45.731961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244765+0000","last_change":"2024-07-26T06:18:59.241701+0000","last_active":"2024-07-26T06:19:00.244765+0000","last_peered":"2024-07-26T06:19:00.244765+0000","last_clean":"2024-07-26T06:19:00.244765+0000","last_became_active":"2024-07-26T06:18:59.239781+0000","last_became_peered":"2024-07-26T06:18:59.239781+0000","last_unstale":"2024-07-26T06:19:00.244765+0000","last_undegraded":"2024-07-26T06:19:00.244765+0000","last_fullsized":"2024-07-26T06:19:00.244765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:41:06.463369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252246+0000","last_change":"2024-07-26T06:18:59.249540+0000","last_active":"2024-07-26T06:18:59.252246+0000","last_peered":"2024-07-26T06:18:59.252246+0000","last_clean":"2024-07-26T06:18:59.252246+0000","last_became_active":"2024-07-26T06:18:59.249195+0000","last_became_peered":"2024-07-26T06:18:59.249195+0000","last_unstale":"2024-07-26T06:18:59.252246+0000","last_undegraded":"2024-07-26T06:18:59.252246+0000","last_fullsized":"2024-07-26T06:18:59.252246+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:02:42.433791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243852+0000","last_change":"2024-07-26T06:18:59.256536+0000","last_active":"2024-07-26T06:19:00.243852+0000","last_peered":"2024-07-26T06:19:00.243852+0000","last_clean":"2024-07-26T06:19:00.243852+0000","last_became_active":"2024-07-26T06:18:59.256275+0000","last_became_peered":"2024-07-26T06:18:59.256275+0000","last_unstale":"2024-07-26T06:19:00.243852+0000","last_undegraded":"2024-07-26T06:19:00.243852+0000","last_fullsized":"2024-07-26T06:19:00.243852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:07.775693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244992+0000","last_change":"2024-07-26T06:18:59.245583+0000","last_active":"2024-07-26T06:19:00.244992+0000","last_peered":"2024-07-26T06:19:00.244992+0000","last_clean":"2024-07-26T06:19:00.244992+0000","last_became_active":"2024-07-26T06:18:59.244895+0000","last_became_peered":"2024-07-26T06:18:59.244895+0000","last_unstale":"2024-07-26T06:19:00.244992+0000","last_undegraded":"2024-07-26T06:19:00.244992+0000","last_fullsized":"2024-07-26T06:19:00.244992+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:11:32.476137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.253237+0000","last_change":"2024-07-26T06:18:59.250553+0000","last_active":"2024-07-26T06:18:59.253237+0000","last_peered":"2024-07-26T06:18:59.253237+0000","last_clean":"2024-07-26T06:18:59.253237+0000","last_became_active":"2024-07-26T06:18:59.248806+0000","last_became_peered":"2024-07-26T06:18:59.248806+0000","last_unstale":"2024-07-26T06:18:59.253237+0000","last_undegraded":"2024-07-26T06:18:59.253237+0000","last_fullsized":"2024-07-26T06:18:59.253237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:47:48.276723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242985+0000","last_change":"2024-07-26T06:18:59.240660+0000","last_active":"2024-07-26T06:19:00.242985+0000","last_peered":"2024-07-26T06:19:00.242985+0000","last_clean":"2024-07-26T06:19:00.242985+0000","last_became_active":"2024-07-26T06:18:59.240399+0000","last_became_peered":"2024-07-26T06:18:59.240399+0000","last_unstale":"2024-07-26T06:19:00.242985+0000","last_undegraded":"2024-07-26T06:19:00.242985+0000","last_fullsized":"2024-07-26T06:19:00.242985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:34:56.315636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251952+0000","last_change":"2024-07-26T06:18:59.247410+0000","last_active":"2024-07-26T06:18:59.251952+0000","last_peered":"2024-07-26T06:18:59.251952+0000","last_clean":"2024-07-26T06:18:59.251952+0000","last_became_active":"2024-07-26T06:18:59.247139+0000","last_became_peered":"2024-07-26T06:18:59.247139+0000","last_unstale":"2024-07-26T06:18:59.251952+0000","last_undegraded":"2024-07-26T06:18:59.251952+0000","last_fullsized":"2024-07-26T06:18:59.251952+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:37.273017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.236427+0000","last_change":"2024-07-26T06:18:59.229828+0000","last_active":"2024-07-26T06:18:59.236427+0000","last_peered":"2024-07-26T06:18:59.236427+0000","last_clean":"2024-07-26T06:18:59.236427+0000","last_became_active":"2024-07-26T06:18:59.229558+0000","last_became_peered":"2024-07-26T06:18:59.229558+0000","last_unstale":"2024-07-26T06:18:59.236427+0000","last_undegraded":"2024-07-26T06:18:59.236427+0000","last_fullsized":"2024-07-26T06:18:59.236427+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:27:22.825430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"98'4","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.247505+0000","last_change":"2024-07-26T06:18:58.235125+0000","last_active":"2024-07-26T06:18:59.247505+0000","last_peered":"2024-07-26T06:18:59.247505+0000","last_clean":"2024-07-26T06:18:59.247505+0000","last_became_active":"2024-07-26T06:18:58.234837+0000","last_became_peered":"2024-07-26T06:18:58.234837+0000","last_unstale":"2024-07-26T06:18:59.247505+0000","last_undegraded":"2024-07-26T06:18:59.247505+0000","last_fullsized":"2024-07-26T06:18:59.247505+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:19.852181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.250028+0000","last_change":"2024-07-26T06:18:59.246491+0000","last_active":"2024-07-26T06:18:59.250028+0000","last_peered":"2024-07-26T06:18:59.250028+0000","last_clean":"2024-07-26T06:18:59.250028+0000","last_became_active":"2024-07-26T06:18:59.246199+0000","last_became_peered":"2024-07-26T06:18:59.246199+0000","last_unstale":"2024-07-26T06:18:59.250028+0000","last_undegraded":"2024-07-26T06:18:59.250028+0000","last_fullsized":"2024-07-26T06:18:59.250028+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:18:54.678508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251072+0000","last_change":"2024-07-26T06:18:58.234573+0000","last_active":"2024-07-26T06:18:59.251072+0000","last_peered":"2024-07-26T06:18:59.251072+0000","last_clean":"2024-07-26T06:18:59.251072+0000","last_became_active":"2024-07-26T06:18:58.234260+0000","last_became_peered":"2024-07-26T06:18:58.234260+0000","last_unstale":"2024-07-26T06:18:59.251072+0000","last_undegraded":"2024-07-26T06:18:59.251072+0000","last_fullsized":"2024-07-26T06:18:59.251072+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:47:47.261472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251838+0000","last_change":"2024-07-26T06:18:59.243764+0000","last_active":"2024-07-26T06:18:59.251838+0000","last_peered":"2024-07-26T06:18:59.251838+0000","last_clean":"2024-07-26T06:18:59.251838+0000","last_became_active":"2024-07-26T06:18:59.243486+0000","last_became_peered":"2024-07-26T06:18:59.243486+0000","last_unstale":"2024-07-26T06:18:59.251838+0000","last_undegraded":"2024-07-26T06:18:59.251838+0000","last_fullsized":"2024-07-26T06:18:59.251838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:01:24.160773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.241918+0000","last_change":"2024-07-26T06:18:58.230586+0000","last_active":"2024-07-26T06:18:59.241918+0000","last_peered":"2024-07-26T06:18:59.241918+0000","last_clean":"2024-07-26T06:18:59.241918+0000","last_became_active":"2024-07-26T06:18:58.230288+0000","last_became_peered":"2024-07-26T06:18:58.230288+0000","last_unstale":"2024-07-26T06:18:59.241918+0000","last_undegraded":"2024-07-26T06:18:59.241918+0000","last_fullsized":"2024-07-26T06:18:59.241918+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:36:47.294190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255826+0000","last_change":"2024-07-26T06:18:59.254060+0000","last_active":"2024-07-26T06:18:59.255826+0000","last_peered":"2024-07-26T06:18:59.255826+0000","last_clean":"2024-07-26T06:18:59.255826+0000","last_became_active":"2024-07-26T06:18:59.251052+0000","last_became_peered":"2024-07-26T06:18:59.251052+0000","last_unstale":"2024-07-26T06:18:59.255826+0000","last_undegraded":"2024-07-26T06:18:59.255826+0000","last_fullsized":"2024-07-26T06:18:59.255826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:32:42.090016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.239240+0000","last_change":"2024-07-26T06:18:58.232832+0000","last_active":"2024-07-26T06:18:59.239240+0000","last_peered":"2024-07-26T06:18:59.239240+0000","last_clean":"2024-07-26T06:18:59.239240+0000","last_became_active":"2024-07-26T06:18:58.228727+0000","last_became_peered":"2024-07-26T06:18:58.228727+0000","last_unstale":"2024-07-26T06:18:59.239240+0000","last_undegraded":"2024-07-26T06:18:59.239240+0000","last_fullsized":"2024-07-26T06:18:59.239240+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:20:49.955403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243468+0000","last_change":"2024-07-26T06:18:59.248460+0000","last_active":"2024-07-26T06:19:00.243468+0000","last_peered":"2024-07-26T06:19:00.243468+0000","last_clean":"2024-07-26T06:19:00.243468+0000","last_became_active":"2024-07-26T06:18:59.248205+0000","last_became_peered":"2024-07-26T06:18:59.248205+0000","last_unstale":"2024-07-26T06:19:00.243468+0000","last_undegraded":"2024-07-26T06:19:00.243468+0000","last_fullsized":"2024-07-26T06:19:00.243468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:12.078801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.243034+0000","last_change":"2024-07-26T06:18:58.242210+0000","last_active":"2024-07-26T06:18:59.243034+0000","last_peered":"2024-07-26T06:18:59.243034+0000","last_clean":"2024-07-26T06:18:59.243034+0000","last_became_active":"2024-07-26T06:18:58.239481+0000","last_became_peered":"2024-07-26T06:18:58.239481+0000","last_unstale":"2024-07-26T06:18:59.243034+0000","last_undegraded":"2024-07-26T06:18:59.243034+0000","last_fullsized":"2024-07-26T06:18:59.243034+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.137519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245226+0000","last_change":"2024-07-26T06:18:59.232001+0000","last_active":"2024-07-26T06:19:00.245226+0000","last_peered":"2024-07-26T06:19:00.245226+0000","last_clean":"2024-07-26T06:19:00.245226+0000","last_became_active":"2024-07-26T06:18:59.231407+0000","last_became_peered":"2024-07-26T06:18:59.231407+0000","last_unstale":"2024-07-26T06:19:00.245226+0000","last_undegraded":"2024-07-26T06:19:00.245226+0000","last_fullsized":"2024-07-26T06:19:00.245226+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:43:14.886460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244598+0000","last_change":"2024-07-26T06:18:58.234670+0000","last_active":"2024-07-26T06:19:00.244598+0000","last_peered":"2024-07-26T06:19:00.244598+0000","last_clean":"2024-07-26T06:19:00.244598+0000","last_became_active":"2024-07-26T06:18:58.234400+0000","last_became_peered":"2024-07-26T06:18:58.234400+0000","last_unstale":"2024-07-26T06:19:00.244598+0000","last_undegraded":"2024-07-26T06:19:00.244598+0000","last_fullsized":"2024-07-26T06:19:00.244598+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:16.824909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243192+0000","last_change":"2024-07-26T06:18:59.230966+0000","last_active":"2024-07-26T06:19:00.243192+0000","last_peered":"2024-07-26T06:19:00.243192+0000","last_clean":"2024-07-26T06:19:00.243192+0000","last_became_active":"2024-07-26T06:18:59.230675+0000","last_became_peered":"2024-07-26T06:18:59.230675+0000","last_unstale":"2024-07-26T06:19:00.243192+0000","last_undegraded":"2024-07-26T06:19:00.243192+0000","last_fullsized":"2024-07-26T06:19:00.243192+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:31:58.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244827+0000","last_change":"2024-07-26T06:18:58.236666+0000","last_active":"2024-07-26T06:19:00.244827+0000","last_peered":"2024-07-26T06:19:00.244827+0000","last_clean":"2024-07-26T06:19:00.244827+0000","last_became_active":"2024-07-26T06:18:58.236310+0000","last_became_peered":"2024-07-26T06:18:58.236310+0000","last_unstale":"2024-07-26T06:19:00.244827+0000","last_undegraded":"2024-07-26T06:19:00.244827+0000","last_fullsized":"2024-07-26T06:19:00.244827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:08:25.599277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.253220+0000","last_change":"2024-07-26T06:18:59.248641+0000","last_active":"2024-07-26T06:18:59.253220+0000","last_peered":"2024-07-26T06:18:59.253220+0000","last_clean":"2024-07-26T06:18:59.253220+0000","last_became_active":"2024-07-26T06:18:59.248363+0000","last_became_peered":"2024-07-26T06:18:59.248363+0000","last_unstale":"2024-07-26T06:18:59.253220+0000","last_undegraded":"2024-07-26T06:18:59.253220+0000","last_fullsized":"2024-07-26T06:18:59.253220+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:50.467113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242901+0000","last_change":"2024-07-26T06:18:58.240064+0000","last_active":"2024-07-26T06:19:00.242901+0000","last_peered":"2024-07-26T06:19:00.242901+0000","last_clean":"2024-07-26T06:19:00.242901+0000","last_became_active":"2024-07-26T06:18:58.239535+0000","last_became_peered":"2024-07-26T06:18:58.239535+0000","last_unstale":"2024-07-26T06:19:00.242901+0000","last_undegraded":"2024-07-26T06:19:00.242901+0000","last_fullsized":"2024-07-26T06:19:00.242901+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:07.330988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.245535+0000","last_change":"2024-07-26T06:18:59.242992+0000","last_active":"2024-07-26T06:18:59.245535+0000","last_peered":"2024-07-26T06:18:59.245535+0000","last_clean":"2024-07-26T06:18:59.245535+0000","last_became_active":"2024-07-26T06:18:59.242687+0000","last_became_peered":"2024-07-26T06:18:59.242687+0000","last_unstale":"2024-07-26T06:18:59.245535+0000","last_undegraded":"2024-07-26T06:18:59.245535+0000","last_fullsized":"2024-07-26T06:18:59.245535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:58.939174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.222694+0000","last_change":"2024-07-26T06:18:58.229299+0000","last_active":"2024-07-26T06:18:59.222694+0000","last_peered":"2024-07-26T06:18:59.222694+0000","last_clean":"2024-07-26T06:18:59.222694+0000","last_became_active":"2024-07-26T06:18:58.229002+0000","last_became_peered":"2024-07-26T06:18:58.229002+0000","last_unstale":"2024-07-26T06:18:59.222694+0000","last_undegraded":"2024-07-26T06:18:59.222694+0000","last_fullsized":"2024-07-26T06:18:59.222694+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:26:35.242032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.250006+0000","last_change":"2024-07-26T06:18:59.245708+0000","last_active":"2024-07-26T06:18:59.250006+0000","last_peered":"2024-07-26T06:18:59.250006+0000","last_clean":"2024-07-26T06:18:59.250006+0000","last_became_active":"2024-07-26T06:18:59.244452+0000","last_became_peered":"2024-07-26T06:18:59.244452+0000","last_unstale":"2024-07-26T06:18:59.250006+0000","last_undegraded":"2024-07-26T06:18:59.250006+0000","last_fullsized":"2024-07-26T06:18:59.250006+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:02.522586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242861+0000","last_change":"2024-07-26T06:18:58.239190+0000","last_active":"2024-07-26T06:19:00.242861+0000","last_peered":"2024-07-26T06:19:00.242861+0000","last_clean":"2024-07-26T06:19:00.242861+0000","last_became_active":"2024-07-26T06:18:58.238927+0000","last_became_peered":"2024-07-26T06:18:58.238927+0000","last_unstale":"2024-07-26T06:19:00.242861+0000","last_undegraded":"2024-07-26T06:19:00.242861+0000","last_fullsized":"2024-07-26T06:19:00.242861+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:43:13.871209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244572+0000","last_change":"2024-07-26T06:18:59.248455+0000","last_active":"2024-07-26T06:19:00.244572+0000","last_peered":"2024-07-26T06:19:00.244572+0000","last_clean":"2024-07-26T06:19:00.244572+0000","last_became_active":"2024-07-26T06:18:59.245481+0000","last_became_peered":"2024-07-26T06:18:59.245481+0000","last_unstale":"2024-07-26T06:19:00.244572+0000","last_undegraded":"2024-07-26T06:19:00.244572+0000","last_fullsized":"2024-07-26T06:19:00.244572+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:43.755259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"98'3","reported_seq":17,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244061+0000","last_change":"2024-07-26T06:18:58.240696+0000","last_active":"2024-07-26T06:19:00.244061+0000","last_peered":"2024-07-26T06:19:00.244061+0000","last_clean":"2024-07-26T06:19:00.244061+0000","last_became_active":"2024-07-26T06:18:58.240340+0000","last_became_peered":"2024-07-26T06:18:58.240340+0000","last_unstale":"2024-07-26T06:19:00.244061+0000","last_undegraded":"2024-07-26T06:19:00.244061+0000","last_fullsized":"2024-07-26T06:19:00.244061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:50.871302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252545+0000","last_change":"2024-07-26T06:18:59.247262+0000","last_active":"2024-07-26T06:18:59.252545+0000","last_peered":"2024-07-26T06:18:59.252545+0000","last_clean":"2024-07-26T06:18:59.252545+0000","last_became_active":"2024-07-26T06:18:59.246993+0000","last_became_peered":"2024-07-26T06:18:59.246993+0000","last_unstale":"2024-07-26T06:18:59.252545+0000","last_undegraded":"2024-07-26T06:18:59.252545+0000","last_fullsized":"2024-07-26T06:18:59.252545+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:55:44.943255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"98'6","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244515+0000","last_change":"2024-07-26T06:18:58.241636+0000","last_active":"2024-07-26T06:19:00.244515+0000","last_peered":"2024-07-26T06:19:00.244515+0000","last_clean":"2024-07-26T06:19:00.244515+0000","last_became_active":"2024-07-26T06:18:58.240783+0000","last_became_peered":"2024-07-26T06:18:58.240783+0000","last_unstale":"2024-07-26T06:19:00.244515+0000","last_undegraded":"2024-07-26T06:19:00.244515+0000","last_fullsized":"2024-07-26T06:19:00.244515+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:29.173157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244538+0000","last_change":"2024-07-26T06:18:59.246256+0000","last_active":"2024-07-26T06:19:00.244538+0000","last_peered":"2024-07-26T06:19:00.244538+0000","last_clean":"2024-07-26T06:19:00.244538+0000","last_became_active":"2024-07-26T06:18:59.245959+0000","last_became_peered":"2024-07-26T06:18:59.245959+0000","last_unstale":"2024-07-26T06:19:00.244538+0000","last_undegraded":"2024-07-26T06:19:00.244538+0000","last_fullsized":"2024-07-26T06:19:00.244538+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:39.726779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.246878+0000","last_change":"2024-07-26T06:18:58.238822+0000","last_active":"2024-07-26T06:18:59.246878+0000","last_peered":"2024-07-26T06:18:59.246878+0000","last_clean":"2024-07-26T06:18:59.246878+0000","last_became_active":"2024-07-26T06:18:58.238445+0000","last_became_peered":"2024-07-26T06:18:58.238445+0000","last_unstale":"2024-07-26T06:18:59.246878+0000","last_undegraded":"2024-07-26T06:18:59.246878+0000","last_fullsized":"2024-07-26T06:18:59.246878+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:19:36.952198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243408+0000","last_change":"2024-07-26T06:18:59.245625+0000","last_active":"2024-07-26T06:19:00.243408+0000","last_peered":"2024-07-26T06:19:00.243408+0000","last_clean":"2024-07-26T06:19:00.243408+0000","last_became_active":"2024-07-26T06:18:59.245367+0000","last_became_peered":"2024-07-26T06:18:59.245367+0000","last_unstale":"2024-07-26T06:19:00.243408+0000","last_undegraded":"2024-07-26T06:19:00.243408+0000","last_fullsized":"2024-07-26T06:19:00.243408+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:46:41.958089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.220982+0000","last_change":"2024-07-26T06:18:58.240538+0000","last_active":"2024-07-26T06:18:59.220982+0000","last_peered":"2024-07-26T06:18:59.220982+0000","last_clean":"2024-07-26T06:18:59.220982+0000","last_became_active":"2024-07-26T06:18:58.240140+0000","last_became_peered":"2024-07-26T06:18:58.240140+0000","last_unstale":"2024-07-26T06:18:59.220982+0000","last_undegraded":"2024-07-26T06:18:59.220982+0000","last_fullsized":"2024-07-26T06:18:59.220982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:28.841009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242822+0000","last_change":"2024-07-26T06:18:59.237309+0000","last_active":"2024-07-26T06:19:00.242822+0000","last_peered":"2024-07-26T06:19:00.242822+0000","last_clean":"2024-07-26T06:19:00.242822+0000","last_became_active":"2024-07-26T06:18:59.236769+0000","last_became_peered":"2024-07-26T06:18:59.236769+0000","last_unstale":"2024-07-26T06:19:00.242822+0000","last_undegraded":"2024-07-26T06:19:00.242822+0000","last_fullsized":"2024-07-26T06:19:00.242822+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:38.135036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"98'8","reported_seq":20,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255125+0000","last_change":"2024-07-26T06:18:58.226227+0000","last_active":"2024-07-26T06:18:59.255125+0000","last_peered":"2024-07-26T06:18:59.255125+0000","last_clean":"2024-07-26T06:18:59.255125+0000","last_became_active":"2024-07-26T06:18:58.225901+0000","last_became_peered":"2024-07-26T06:18:58.225901+0000","last_unstale":"2024-07-26T06:18:59.255125+0000","last_undegraded":"2024-07-26T06:18:59.255125+0000","last_fullsized":"2024-07-26T06:18:59.255125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:28.175721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249756+0000","last_change":"2024-07-26T06:18:59.240235+0000","last_active":"2024-07-26T06:18:59.249756+0000","last_peered":"2024-07-26T06:18:59.249756+0000","last_clean":"2024-07-26T06:18:59.249756+0000","last_became_active":"2024-07-26T06:18:59.239350+0000","last_became_peered":"2024-07-26T06:18:59.239350+0000","last_unstale":"2024-07-26T06:18:59.249756+0000","last_undegraded":"2024-07-26T06:18:59.249756+0000","last_fullsized":"2024-07-26T06:18:59.249756+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.098824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245497+0000","last_change":"2024-07-26T06:18:58.235735+0000","last_active":"2024-07-26T06:19:00.245497+0000","last_peered":"2024-07-26T06:19:00.245497+0000","last_clean":"2024-07-26T06:19:00.245497+0000","last_became_active":"2024-07-26T06:18:58.235305+0000","last_became_peered":"2024-07-26T06:18:58.235305+0000","last_unstale":"2024-07-26T06:19:00.245497+0000","last_undegraded":"2024-07-26T06:19:00.245497+0000","last_fullsized":"2024-07-26T06:19:00.245497+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:41:35.858030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249664+0000","last_change":"2024-07-26T06:18:59.242674+0000","last_active":"2024-07-26T06:18:59.249664+0000","last_peered":"2024-07-26T06:18:59.249664+0000","last_clean":"2024-07-26T06:18:59.249664+0000","last_became_active":"2024-07-26T06:18:59.242280+0000","last_became_peered":"2024-07-26T06:18:59.242280+0000","last_unstale":"2024-07-26T06:18:59.249664+0000","last_undegraded":"2024-07-26T06:18:59.249664+0000","last_fullsized":"2024-07-26T06:18:59.249664+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:54:19.159552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249700+0000","last_change":"2024-07-26T06:18:58.230026+0000","last_active":"2024-07-26T06:18:59.249700+0000","last_peered":"2024-07-26T06:18:59.249700+0000","last_clean":"2024-07-26T06:18:59.249700+0000","last_became_active":"2024-07-26T06:18:58.229730+0000","last_became_peered":"2024-07-26T06:18:58.229730+0000","last_unstale":"2024-07-26T06:18:59.249700+0000","last_undegraded":"2024-07-26T06:18:59.249700+0000","last_fullsized":"2024-07-26T06:18:59.249700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:32:49.133605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.260650+0000","last_change":"2024-07-26T06:18:59.257176+0000","last_active":"2024-07-26T06:18:59.260650+0000","last_peered":"2024-07-26T06:18:59.260650+0000","last_clean":"2024-07-26T06:18:59.260650+0000","last_became_active":"2024-07-26T06:18:59.256831+0000","last_became_peered":"2024-07-26T06:18:59.256831+0000","last_unstale":"2024-07-26T06:18:59.260650+0000","last_undegraded":"2024-07-26T06:18:59.260650+0000","last_fullsized":"2024-07-26T06:18:59.260650+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:02.107634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.246099+0000","last_change":"2024-07-26T06:18:58.241191+0000","last_active":"2024-07-26T06:18:59.246099+0000","last_peered":"2024-07-26T06:18:59.246099+0000","last_clean":"2024-07-26T06:18:59.246099+0000","last_became_active":"2024-07-26T06:18:58.241027+0000","last_became_peered":"2024-07-26T06:18:58.241027+0000","last_unstale":"2024-07-26T06:18:59.246099+0000","last_undegraded":"2024-07-26T06:18:59.246099+0000","last_fullsized":"2024-07-26T06:18:59.246099+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:12.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.256334+0000","last_change":"2024-07-26T06:18:59.254766+0000","last_active":"2024-07-26T06:18:59.256334+0000","last_peered":"2024-07-26T06:18:59.256334+0000","last_clean":"2024-07-26T06:18:59.256334+0000","last_became_active":"2024-07-26T06:18:59.254460+0000","last_became_peered":"2024-07-26T06:18:59.254460+0000","last_unstale":"2024-07-26T06:18:59.256334+0000","last_undegraded":"2024-07-26T06:18:59.256334+0000","last_fullsized":"2024-07-26T06:18:59.256334+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:47:11.658319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.233594+0000","last_change":"2024-07-26T06:18:58.240084+0000","last_active":"2024-07-26T06:18:59.233594+0000","last_peered":"2024-07-26T06:18:59.233594+0000","last_clean":"2024-07-26T06:18:59.233594+0000","last_became_active":"2024-07-26T06:18:58.239739+0000","last_became_peered":"2024-07-26T06:18:58.239739+0000","last_unstale":"2024-07-26T06:18:59.233594+0000","last_undegraded":"2024-07-26T06:18:59.233594+0000","last_fullsized":"2024-07-26T06:18:59.233594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:59.327981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252826+0000","last_change":"2024-07-26T06:18:59.247942+0000","last_active":"2024-07-26T06:18:59.252826+0000","last_peered":"2024-07-26T06:18:59.252826+0000","last_clean":"2024-07-26T06:18:59.252826+0000","last_became_active":"2024-07-26T06:18:59.247669+0000","last_became_peered":"2024-07-26T06:18:59.247669+0000","last_unstale":"2024-07-26T06:18:59.252826+0000","last_undegraded":"2024-07-26T06:18:59.252826+0000","last_fullsized":"2024-07-26T06:18:59.252826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:35.380377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.239900+0000","last_change":"2024-07-26T06:18:58.236979+0000","last_active":"2024-07-26T06:18:59.239900+0000","last_peered":"2024-07-26T06:18:59.239900+0000","last_clean":"2024-07-26T06:18:59.239900+0000","last_became_active":"2024-07-26T06:18:58.236704+0000","last_became_peered":"2024-07-26T06:18:58.236704+0000","last_unstale":"2024-07-26T06:18:59.239900+0000","last_undegraded":"2024-07-26T06:18:59.239900+0000","last_fullsized":"2024-07-26T06:18:59.239900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:42:10.751849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243941+0000","last_change":"2024-07-26T06:18:59.244713+0000","last_active":"2024-07-26T06:19:00.243941+0000","last_peered":"2024-07-26T06:19:00.243941+0000","last_clean":"2024-07-26T06:19:00.243941+0000","last_became_active":"2024-07-26T06:18:59.244331+0000","last_became_peered":"2024-07-26T06:18:59.244331+0000","last_unstale":"2024-07-26T06:19:00.243941+0000","last_undegraded":"2024-07-26T06:19:00.243941+0000","last_fullsized":"2024-07-26T06:19:00.243941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:11:40.535982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.222583+0000","last_change":"2024-07-26T06:18:58.232007+0000","last_active":"2024-07-26T06:18:59.222583+0000","last_peered":"2024-07-26T06:18:59.222583+0000","last_clean":"2024-07-26T06:18:59.222583+0000","last_became_active":"2024-07-26T06:18:58.231746+0000","last_became_peered":"2024-07-26T06:18:58.231746+0000","last_unstale":"2024-07-26T06:18:59.222583+0000","last_undegraded":"2024-07-26T06:18:59.222583+0000","last_fullsized":"2024-07-26T06:18:59.222583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:05.393787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.254132+0000","last_change":"2024-07-26T06:18:59.252085+0000","last_active":"2024-07-26T06:18:59.254132+0000","last_peered":"2024-07-26T06:18:59.254132+0000","last_clean":"2024-07-26T06:18:59.254132+0000","last_became_active":"2024-07-26T06:18:59.250358+0000","last_became_peered":"2024-07-26T06:18:59.250358+0000","last_unstale":"2024-07-26T06:18:59.254132+0000","last_undegraded":"2024-07-26T06:18:59.254132+0000","last_fullsized":"2024-07-26T06:18:59.254132+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:06.334942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.222035+0000","last_change":"2024-07-26T06:18:58.232662+0000","last_active":"2024-07-26T06:18:59.222035+0000","last_peered":"2024-07-26T06:18:59.222035+0000","last_clean":"2024-07-26T06:18:59.222035+0000","last_became_active":"2024-07-26T06:18:58.232338+0000","last_became_peered":"2024-07-26T06:18:58.232338+0000","last_unstale":"2024-07-26T06:18:59.222035+0000","last_undegraded":"2024-07-26T06:18:59.222035+0000","last_fullsized":"2024-07-26T06:18:59.222035+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:03:41.924302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242985+0000","last_change":"2024-07-26T06:18:59.257900+0000","last_active":"2024-07-26T06:19:00.242985+0000","last_peered":"2024-07-26T06:19:00.242985+0000","last_clean":"2024-07-26T06:19:00.242985+0000","last_became_active":"2024-07-26T06:18:59.257599+0000","last_became_peered":"2024-07-26T06:18:59.257599+0000","last_unstale":"2024-07-26T06:19:00.242985+0000","last_undegraded":"2024-07-26T06:19:00.242985+0000","last_fullsized":"2024-07-26T06:19:00.242985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:20:03.140222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243046+0000","last_change":"2024-07-26T06:18:58.238882+0000","last_active":"2024-07-26T06:19:00.243046+0000","last_peered":"2024-07-26T06:19:00.243046+0000","last_clean":"2024-07-26T06:19:00.243046+0000","last_became_active":"2024-07-26T06:18:58.235910+0000","last_became_peered":"2024-07-26T06:18:58.235910+0000","last_unstale":"2024-07-26T06:19:00.243046+0000","last_undegraded":"2024-07-26T06:19:00.243046+0000","last_fullsized":"2024-07-26T06:19:00.243046+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:19.617245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249198+0000","last_change":"2024-07-26T06:18:59.244015+0000","last_active":"2024-07-26T06:18:59.249198+0000","last_peered":"2024-07-26T06:18:59.249198+0000","last_clean":"2024-07-26T06:18:59.249198+0000","last_became_active":"2024-07-26T06:18:59.243739+0000","last_became_peered":"2024-07-26T06:18:59.243739+0000","last_unstale":"2024-07-26T06:18:59.249198+0000","last_undegraded":"2024-07-26T06:18:59.249198+0000","last_fullsized":"2024-07-26T06:18:59.249198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:05:42.409469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243270+0000","last_change":"2024-07-26T06:18:58.240969+0000","last_active":"2024-07-26T06:19:00.243270+0000","last_peered":"2024-07-26T06:19:00.243270+0000","last_clean":"2024-07-26T06:19:00.243270+0000","last_became_active":"2024-07-26T06:18:58.240652+0000","last_became_peered":"2024-07-26T06:18:58.240652+0000","last_unstale":"2024-07-26T06:19:00.243270+0000","last_undegraded":"2024-07-26T06:19:00.243270+0000","last_fullsized":"2024-07-26T06:19:00.243270+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:51.735453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245533+0000","last_change":"2024-07-26T06:18:59.236493+0000","last_active":"2024-07-26T06:19:00.245533+0000","last_peered":"2024-07-26T06:19:00.245533+0000","last_clean":"2024-07-26T06:19:00.245533+0000","last_became_active":"2024-07-26T06:18:59.233060+0000","last_became_peered":"2024-07-26T06:18:59.233060+0000","last_unstale":"2024-07-26T06:19:00.245533+0000","last_undegraded":"2024-07-26T06:19:00.245533+0000","last_fullsized":"2024-07-26T06:19:00.245533+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:22:38.466217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.227386+0000","last_change":"2024-07-26T06:18:58.233524+0000","last_active":"2024-07-26T06:18:59.227386+0000","last_peered":"2024-07-26T06:18:59.227386+0000","last_clean":"2024-07-26T06:18:59.227386+0000","last_became_active":"2024-07-26T06:18:58.233181+0000","last_became_peered":"2024-07-26T06:18:58.233181+0000","last_unstale":"2024-07-26T06:18:59.227386+0000","last_undegraded":"2024-07-26T06:18:59.227386+0000","last_fullsized":"2024-07-26T06:18:59.227386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:55:32.703489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244374+0000","last_change":"2024-07-26T06:18:59.230457+0000","last_active":"2024-07-26T06:19:00.244374+0000","last_peered":"2024-07-26T06:19:00.244374+0000","last_clean":"2024-07-26T06:19:00.244374+0000","last_became_active":"2024-07-26T06:18:59.224988+0000","last_became_peered":"2024-07-26T06:18:59.224988+0000","last_unstale":"2024-07-26T06:19:00.244374+0000","last_undegraded":"2024-07-26T06:19:00.244374+0000","last_fullsized":"2024-07-26T06:19:00.244374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:06:56.302690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244194+0000","last_change":"2024-07-26T06:18:58.235818+0000","last_active":"2024-07-26T06:19:00.244194+0000","last_peered":"2024-07-26T06:19:00.244194+0000","last_clean":"2024-07-26T06:19:00.244194+0000","last_became_active":"2024-07-26T06:18:58.235451+0000","last_became_peered":"2024-07-26T06:18:58.235451+0000","last_unstale":"2024-07-26T06:19:00.244194+0000","last_undegraded":"2024-07-26T06:19:00.244194+0000","last_fullsized":"2024-07-26T06:19:00.244194+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:16.797214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.257047+0000","last_change":"2024-07-26T06:18:59.255794+0000","last_active":"2024-07-26T06:18:59.257047+0000","last_peered":"2024-07-26T06:18:59.257047+0000","last_clean":"2024-07-26T06:18:59.257047+0000","last_became_active":"2024-07-26T06:18:59.254994+0000","last_became_peered":"2024-07-26T06:18:59.254994+0000","last_unstale":"2024-07-26T06:18:59.257047+0000","last_undegraded":"2024-07-26T06:18:59.257047+0000","last_fullsized":"2024-07-26T06:18:59.257047+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:08.346239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244482+0000","last_change":"2024-07-26T06:18:58.238066+0000","last_active":"2024-07-26T06:19:00.244482+0000","last_peered":"2024-07-26T06:19:00.244482+0000","last_clean":"2024-07-26T06:19:00.244482+0000","last_became_active":"2024-07-26T06:18:58.235846+0000","last_became_peered":"2024-07-26T06:18:58.235846+0000","last_unstale":"2024-07-26T06:19:00.244482+0000","last_undegraded":"2024-07-26T06:19:00.244482+0000","last_fullsized":"2024-07-26T06:19:00.244482+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:04:52.958647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245165+0000","last_change":"2024-07-26T06:18:59.239981+0000","last_active":"2024-07-26T06:19:00.245165+0000","last_peered":"2024-07-26T06:19:00.245165+0000","last_clean":"2024-07-26T06:19:00.245165+0000","last_became_active":"2024-07-26T06:18:59.239654+0000","last_became_peered":"2024-07-26T06:18:59.239654+0000","last_unstale":"2024-07-26T06:19:00.245165+0000","last_undegraded":"2024-07-26T06:19:00.245165+0000","last_fullsized":"2024-07-26T06:19:00.245165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:52.750704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.229936+0000","last_change":"2024-07-26T06:18:58.232570+0000","last_active":"2024-07-26T06:18:59.229936+0000","last_peered":"2024-07-26T06:18:59.229936+0000","last_clean":"2024-07-26T06:18:59.229936+0000","last_became_active":"2024-07-26T06:18:58.232239+0000","last_became_peered":"2024-07-26T06:18:58.232239+0000","last_unstale":"2024-07-26T06:18:59.229936+0000","last_undegraded":"2024-07-26T06:18:59.229936+0000","last_fullsized":"2024-07-26T06:18:59.229936+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:41:51.104522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245487+0000","last_change":"2024-07-26T06:18:59.239077+0000","last_active":"2024-07-26T06:19:00.245487+0000","last_peered":"2024-07-26T06:19:00.245487+0000","last_clean":"2024-07-26T06:19:00.245487+0000","last_became_active":"2024-07-26T06:18:59.237189+0000","last_became_peered":"2024-07-26T06:18:59.237189+0000","last_unstale":"2024-07-26T06:19:00.245487+0000","last_undegraded":"2024-07-26T06:19:00.245487+0000","last_fullsized":"2024-07-26T06:19:00.245487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:17.840160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245481+0000","last_change":"2024-07-26T06:18:58.236663+0000","last_active":"2024-07-26T06:19:00.245481+0000","last_peered":"2024-07-26T06:19:00.245481+0000","last_clean":"2024-07-26T06:19:00.245481+0000","last_became_active":"2024-07-26T06:18:58.236174+0000","last_became_peered":"2024-07-26T06:18:58.236174+0000","last_unstale":"2024-07-26T06:19:00.245481+0000","last_undegraded":"2024-07-26T06:19:00.245481+0000","last_fullsized":"2024-07-26T06:19:00.245481+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:49.134754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.254091+0000","last_change":"2024-07-26T06:18:59.251090+0000","last_active":"2024-07-26T06:18:59.254091+0000","last_peered":"2024-07-26T06:18:59.254091+0000","last_clean":"2024-07-26T06:18:59.254091+0000","last_became_active":"2024-07-26T06:18:59.250766+0000","last_became_peered":"2024-07-26T06:18:59.250766+0000","last_unstale":"2024-07-26T06:18:59.254091+0000","last_undegraded":"2024-07-26T06:18:59.254091+0000","last_fullsized":"2024-07-26T06:18:59.254091+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:10:24.921556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.223661+0000","last_change":"2024-07-26T06:18:58.231335+0000","last_active":"2024-07-26T06:18:59.223661+0000","last_peered":"2024-07-26T06:18:59.223661+0000","last_clean":"2024-07-26T06:18:59.223661+0000","last_became_active":"2024-07-26T06:18:58.227857+0000","last_became_peered":"2024-07-26T06:18:58.227857+0000","last_unstale":"2024-07-26T06:18:59.223661+0000","last_undegraded":"2024-07-26T06:18:59.223661+0000","last_fullsized":"2024-07-26T06:18:59.223661+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:00.503756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243161+0000","last_change":"2024-07-26T06:18:59.232505+0000","last_active":"2024-07-26T06:19:00.243161+0000","last_peered":"2024-07-26T06:19:00.243161+0000","last_clean":"2024-07-26T06:19:00.243161+0000","last_became_active":"2024-07-26T06:18:59.225519+0000","last_became_peered":"2024-07-26T06:18:59.225519+0000","last_unstale":"2024-07-26T06:19:00.243161+0000","last_undegraded":"2024-07-26T06:19:00.243161+0000","last_fullsized":"2024-07-26T06:19:00.243161+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:05.002802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243413+0000","last_change":"2024-07-26T06:18:58.239275+0000","last_active":"2024-07-26T06:19:00.243413+0000","last_peered":"2024-07-26T06:19:00.243413+0000","last_clean":"2024-07-26T06:19:00.243413+0000","last_became_active":"2024-07-26T06:18:58.235553+0000","last_became_peered":"2024-07-26T06:18:58.235553+0000","last_unstale":"2024-07-26T06:19:00.243413+0000","last_undegraded":"2024-07-26T06:19:00.243413+0000","last_fullsized":"2024-07-26T06:19:00.243413+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:53.015361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244090+0000","last_change":"2024-07-26T06:18:59.247263+0000","last_active":"2024-07-26T06:19:00.244090+0000","last_peered":"2024-07-26T06:19:00.244090+0000","last_clean":"2024-07-26T06:19:00.244090+0000","last_became_active":"2024-07-26T06:18:59.246666+0000","last_became_peered":"2024-07-26T06:18:59.246666+0000","last_unstale":"2024-07-26T06:19:00.244090+0000","last_undegraded":"2024-07-26T06:19:00.244090+0000","last_fullsized":"2024-07-26T06:19:00.244090+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:31.569170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.231831+0000","last_change":"2024-07-26T06:18:58.240401+0000","last_active":"2024-07-26T06:18:59.231831+0000","last_peered":"2024-07-26T06:18:59.231831+0000","last_clean":"2024-07-26T06:18:59.231831+0000","last_became_active":"2024-07-26T06:18:58.240053+0000","last_became_peered":"2024-07-26T06:18:58.240053+0000","last_unstale":"2024-07-26T06:18:59.231831+0000","last_undegraded":"2024-07-26T06:18:59.231831+0000","last_fullsized":"2024-07-26T06:18:59.231831+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:51.142931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'79","reported_seq":278,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.238624+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:18:59.238624+0000","last_peered":"2024-07-26T06:18:59.238624+0000","last_clean":"2024-07-26T06:18:59.238624+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:18:59.238624+0000","last_undegraded":"2024-07-26T06:18:59.238624+0000","last_fullsized":"2024-07-26T06:18:59.238624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509511,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32960,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":1814528,"data_stored":1119423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509512,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":528509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1220608,"data_stored":529043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542215,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1212416,"data_stored":528439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1228800,"data_stored":530121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":528439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542216,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1220608,"data_stored":528563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509512,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1120531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:19:03.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 -- 172.21.15.163:0/3688065605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c6806bf40 msgr2=0x7f5c6806c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 --2- 172.21.15.163:0/3688065605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c6806bf40 0x7f5c6806c320 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5c68070bc0 tx=0x7f5c6002a790 comp rx=0 tx=0).stop 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 -- 172.21.15.163:0/3688065605 shutdown_connections 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 --2- 172.21.15.163:0/3688065605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c6806bf40 0x7f5c6806c320 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 --2- 172.21.15.163:0/3688065605 >> v2:172.21.15.164:3300/0 conn(0x7f5c680cb7e0 0x7f5c6815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 -- 172.21.15.163:0/3688065605 >> 172.21.15.163:0/3688065605 conn(0x7f5c6806d420 msgr2=0x7f5c6806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 -- 172.21.15.163:0/3688065605 shutdown_connections 2024-07-26T06:19:03.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.935+0000 7f5c70000640 1 -- 172.21.15.163:0/3688065605 wait complete. 2024-07-26T06:19:03.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.937+0000 7f5c70000640 1 Processor -- start 2024-07-26T06:19:03.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.937+0000 7f5c70000640 1 -- start start 2024-07-26T06:19:03.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c70000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c6806bf40 0x7f5c68153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c70000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c68071cf0 0x7f5c68156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c70000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 0x7f5c6815a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c70000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c680736b0 con 0x7f5c680cb7e0 2024-07-26T06:19:03.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c70000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c68073530 con 0x7f5c68071cf0 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c70000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c68073830 con 0x7f5c6806bf40 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 0x7f5c6815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 0x7f5c6815a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35916/0 (socket says 172.21.15.163:35916) 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 -- 172.21.15.163:0/3024441492 learned_addr learned my addr 172.21.15.163:0/3024441492 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6d600640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c6806bf40 0x7f5c68153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 -- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c6806bf40 msgr2=0x7f5c68153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c6806bf40 0x7f5c68153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 -- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c68071cf0 msgr2=0x7f5c68156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c68071cf0 0x7f5c68156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.938+0000 7f5c6e000640 1 -- 172.21.15.163:0/3024441492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c68157620 con 0x7f5c680cb7e0 2024-07-26T06:19:03.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.939+0000 7f5c6e000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 0x7f5c6815a230 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5c60034040 tx=0x7f5c60003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:03.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.939+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60041020 con 0x7f5c680cb7e0 2024-07-26T06:19:03.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.939+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c60004330 con 0x7f5c680cb7e0 2024-07-26T06:19:03.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.939+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60036040 con 0x7f5c680cb7e0 2024-07-26T06:19:03.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.939+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c68157cb0 con 0x7f5c680cb7e0 2024-07-26T06:19:03.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.939+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c68154060 con 0x7f5c680cb7e0 2024-07-26T06:19:03.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.941+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c68073d00 con 0x7f5c680cb7e0 2024-07-26T06:19:03.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.942+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f5c60044020 con 0x7f5c680cb7e0 2024-07-26T06:19:03.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.943+0000 7f5c65600640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5c480763f0 0x7f5c480788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:03.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.943+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7f5c600b8c30 con 0x7f5c680cb7e0 2024-07-26T06:19:03.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.943+0000 7f5c6d600640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5c480763f0 0x7f5c480788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:03.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.944+0000 7f5c6d600640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5c480763f0 0x7f5c480788b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0058c0 tx=0x7f5c5c005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:03.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:03.947+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5c6004b020 con 0x7f5c680cb7e0 2024-07-26T06:19:04.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.149+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c68074040 con 0x7f5c480763f0 2024-07-26T06:19:04.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.154+0000 7f5c65600640 1 -- 172.21.15.163:0/3024441492 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252038 (secure 0 0 0) 0x7f5c68074040 con 0x7f5c480763f0 2024-07-26T06:19:04.159 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:04.160 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5c480763f0 msgr2=0x7f5c480788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5c480763f0 0x7f5c480788b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0058c0 tx=0x7f5c5c005850 comp rx=0 tx=0).stop 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 msgr2=0x7f5c6815a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 0x7f5c6815a230 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5c60034040 tx=0x7f5c60003f10 comp rx=0 tx=0).stop 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 shutdown_connections 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f5c480763f0 0x7f5c480788b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c680cb7e0 0x7f5c6815a230 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c68071cf0 0x7f5c68156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 --2- 172.21.15.163:0/3024441492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c6806bf40 0x7f5c68153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 >> 172.21.15.163:0/3024441492 conn(0x7f5c6806d420 msgr2=0x7f5c6806d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:04.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 shutdown_connections 2024-07-26T06:19:04.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.160+0000 7f5c70000640 1 -- 172.21.15.163:0/3024441492 wait complete. 2024-07-26T06:19:04.180 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-07-26T06:19:02.474754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260188,"kb_used_data":10696,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714532,"statfs":{"total":773094113280,"available":772827680768,"internally_reserved":0,"allocated":10952704,"data_stored":5413068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.288971"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245385+0000","last_change":"2024-07-26T06:18:59.243519+0000","last_active":"2024-07-26T06:19:00.245385+0000","last_peered":"2024-07-26T06:19:00.245385+0000","last_clean":"2024-07-26T06:19:00.245385+0000","last_became_active":"2024-07-26T06:18:59.235918+0000","last_became_peered":"2024-07-26T06:18:59.235918+0000","last_unstale":"2024-07-26T06:19:00.245385+0000","last_undegraded":"2024-07-26T06:19:00.245385+0000","last_fullsized":"2024-07-26T06:19:00.245385+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:27:54.756486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.246490+0000","last_change":"2024-07-26T06:18:59.240213+0000","last_active":"2024-07-26T06:19:00.246490+0000","last_peered":"2024-07-26T06:19:00.246490+0000","last_clean":"2024-07-26T06:19:00.246490+0000","last_became_active":"2024-07-26T06:18:59.237248+0000","last_became_peered":"2024-07-26T06:18:59.237248+0000","last_unstale":"2024-07-26T06:19:00.246490+0000","last_undegraded":"2024-07-26T06:19:00.246490+0000","last_fullsized":"2024-07-26T06:19:00.246490+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:48.869491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255767+0000","last_change":"2024-07-26T06:18:59.248831+0000","last_active":"2024-07-26T06:18:59.255767+0000","last_peered":"2024-07-26T06:18:59.255767+0000","last_clean":"2024-07-26T06:18:59.255767+0000","last_became_active":"2024-07-26T06:18:59.248572+0000","last_became_peered":"2024-07-26T06:18:59.248572+0000","last_unstale":"2024-07-26T06:18:59.255767+0000","last_undegraded":"2024-07-26T06:18:59.255767+0000","last_fullsized":"2024-07-26T06:18:59.255767+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:33.396453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244106+0000","last_change":"2024-07-26T06:18:59.236444+0000","last_active":"2024-07-26T06:19:00.244106+0000","last_peered":"2024-07-26T06:19:00.244106+0000","last_clean":"2024-07-26T06:19:00.244106+0000","last_became_active":"2024-07-26T06:18:59.234730+0000","last_became_peered":"2024-07-26T06:18:59.234730+0000","last_unstale":"2024-07-26T06:19:00.244106+0000","last_undegraded":"2024-07-26T06:19:00.244106+0000","last_fullsized":"2024-07-26T06:19:00.244106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:02:02.418506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251244+0000","last_change":"2024-07-26T06:18:59.248389+0000","last_active":"2024-07-26T06:18:59.251244+0000","last_peered":"2024-07-26T06:18:59.251244+0000","last_clean":"2024-07-26T06:18:59.251244+0000","last_became_active":"2024-07-26T06:18:59.248072+0000","last_became_peered":"2024-07-26T06:18:59.248072+0000","last_unstale":"2024-07-26T06:18:59.251244+0000","last_undegraded":"2024-07-26T06:18:59.251244+0000","last_fullsized":"2024-07-26T06:18:59.251244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:45:40.679563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251922+0000","last_change":"2024-07-26T06:18:59.243256+0000","last_active":"2024-07-26T06:18:59.251922+0000","last_peered":"2024-07-26T06:18:59.251922+0000","last_clean":"2024-07-26T06:18:59.251922+0000","last_became_active":"2024-07-26T06:18:59.242688+0000","last_became_peered":"2024-07-26T06:18:59.242688+0000","last_unstale":"2024-07-26T06:18:59.251922+0000","last_undegraded":"2024-07-26T06:18:59.251922+0000","last_fullsized":"2024-07-26T06:18:59.251922+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:21:43.549583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.253243+0000","last_change":"2024-07-26T06:18:59.248843+0000","last_active":"2024-07-26T06:18:59.253243+0000","last_peered":"2024-07-26T06:18:59.253243+0000","last_clean":"2024-07-26T06:18:59.253243+0000","last_became_active":"2024-07-26T06:18:59.248300+0000","last_became_peered":"2024-07-26T06:18:59.248300+0000","last_unstale":"2024-07-26T06:18:59.253243+0000","last_undegraded":"2024-07-26T06:18:59.253243+0000","last_fullsized":"2024-07-26T06:18:59.253243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:44:50.902450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.256488+0000","last_change":"2024-07-26T06:18:59.255038+0000","last_active":"2024-07-26T06:18:59.256488+0000","last_peered":"2024-07-26T06:18:59.256488+0000","last_clean":"2024-07-26T06:18:59.256488+0000","last_became_active":"2024-07-26T06:18:59.254167+0000","last_became_peered":"2024-07-26T06:18:59.254167+0000","last_unstale":"2024-07-26T06:18:59.256488+0000","last_undegraded":"2024-07-26T06:18:59.256488+0000","last_fullsized":"2024-07-26T06:18:59.256488+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:29.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245385+0000","last_change":"2024-07-26T06:18:59.247197+0000","last_active":"2024-07-26T06:19:00.245385+0000","last_peered":"2024-07-26T06:19:00.245385+0000","last_clean":"2024-07-26T06:19:00.245385+0000","last_became_active":"2024-07-26T06:18:59.246611+0000","last_became_peered":"2024-07-26T06:18:59.246611+0000","last_unstale":"2024-07-26T06:19:00.245385+0000","last_undegraded":"2024-07-26T06:19:00.245385+0000","last_fullsized":"2024-07-26T06:19:00.245385+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:53:07.319925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244597+0000","last_change":"2024-07-26T06:18:59.231835+0000","last_active":"2024-07-26T06:19:00.244597+0000","last_peered":"2024-07-26T06:19:00.244597+0000","last_clean":"2024-07-26T06:19:00.244597+0000","last_became_active":"2024-07-26T06:18:59.231553+0000","last_became_peered":"2024-07-26T06:18:59.231553+0000","last_unstale":"2024-07-26T06:19:00.244597+0000","last_undegraded":"2024-07-26T06:19:00.244597+0000","last_fullsized":"2024-07-26T06:19:00.244597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:54:44.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.246324+0000","last_change":"2024-07-26T06:18:59.239507+0000","last_active":"2024-07-26T06:19:00.246324+0000","last_peered":"2024-07-26T06:19:00.246324+0000","last_clean":"2024-07-26T06:19:00.246324+0000","last_became_active":"2024-07-26T06:18:59.236138+0000","last_became_peered":"2024-07-26T06:18:59.236138+0000","last_unstale":"2024-07-26T06:19:00.246324+0000","last_undegraded":"2024-07-26T06:19:00.246324+0000","last_fullsized":"2024-07-26T06:19:00.246324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:12:29.396937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244702+0000","last_change":"2024-07-26T06:18:59.231512+0000","last_active":"2024-07-26T06:19:00.244702+0000","last_peered":"2024-07-26T06:19:00.244702+0000","last_clean":"2024-07-26T06:19:00.244702+0000","last_became_active":"2024-07-26T06:18:59.231141+0000","last_became_peered":"2024-07-26T06:18:59.231141+0000","last_unstale":"2024-07-26T06:19:00.244702+0000","last_undegraded":"2024-07-26T06:19:00.244702+0000","last_fullsized":"2024-07-26T06:19:00.244702+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:01:58.911434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.257520+0000","last_change":"2024-07-26T06:18:59.256066+0000","last_active":"2024-07-26T06:18:59.257520+0000","last_peered":"2024-07-26T06:18:59.257520+0000","last_clean":"2024-07-26T06:18:59.257520+0000","last_became_active":"2024-07-26T06:18:59.255754+0000","last_became_peered":"2024-07-26T06:18:59.255754+0000","last_unstale":"2024-07-26T06:18:59.257520+0000","last_undegraded":"2024-07-26T06:18:59.257520+0000","last_fullsized":"2024-07-26T06:18:59.257520+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:55:33.718740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255990+0000","last_change":"2024-07-26T06:18:59.252936+0000","last_active":"2024-07-26T06:18:59.255990+0000","last_peered":"2024-07-26T06:18:59.255990+0000","last_clean":"2024-07-26T06:18:59.255990+0000","last_became_active":"2024-07-26T06:18:59.252571+0000","last_became_peered":"2024-07-26T06:18:59.252571+0000","last_unstale":"2024-07-26T06:18:59.255990+0000","last_undegraded":"2024-07-26T06:18:59.255990+0000","last_fullsized":"2024-07-26T06:18:59.255990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:28:56.774761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252007+0000","last_change":"2024-07-26T06:18:59.248483+0000","last_active":"2024-07-26T06:18:59.252007+0000","last_peered":"2024-07-26T06:18:59.252007+0000","last_clean":"2024-07-26T06:18:59.252007+0000","last_became_active":"2024-07-26T06:18:59.248218+0000","last_became_peered":"2024-07-26T06:18:59.248218+0000","last_unstale":"2024-07-26T06:18:59.252007+0000","last_undegraded":"2024-07-26T06:18:59.252007+0000","last_fullsized":"2024-07-26T06:18:59.252007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:53:28.329133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242626+0000","last_change":"2024-07-26T06:18:59.244881+0000","last_active":"2024-07-26T06:19:00.242626+0000","last_peered":"2024-07-26T06:19:00.242626+0000","last_clean":"2024-07-26T06:19:00.242626+0000","last_became_active":"2024-07-26T06:18:59.244485+0000","last_became_peered":"2024-07-26T06:18:59.244485+0000","last_unstale":"2024-07-26T06:19:00.242626+0000","last_undegraded":"2024-07-26T06:19:00.242626+0000","last_fullsized":"2024-07-26T06:19:00.242626+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:26.825732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.246621+0000","last_change":"2024-07-26T06:18:59.236328+0000","last_active":"2024-07-26T06:19:00.246621+0000","last_peered":"2024-07-26T06:19:00.246621+0000","last_clean":"2024-07-26T06:19:00.246621+0000","last_became_active":"2024-07-26T06:18:59.235877+0000","last_became_peered":"2024-07-26T06:18:59.235877+0000","last_unstale":"2024-07-26T06:19:00.246621+0000","last_undegraded":"2024-07-26T06:19:00.246621+0000","last_fullsized":"2024-07-26T06:19:00.246621+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:48:38.222129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.247696+0000","last_change":"2024-07-26T06:18:59.240906+0000","last_active":"2024-07-26T06:18:59.247696+0000","last_peered":"2024-07-26T06:18:59.247696+0000","last_clean":"2024-07-26T06:18:59.247696+0000","last_became_active":"2024-07-26T06:18:59.240643+0000","last_became_peered":"2024-07-26T06:18:59.240643+0000","last_unstale":"2024-07-26T06:18:59.247696+0000","last_undegraded":"2024-07-26T06:18:59.247696+0000","last_fullsized":"2024-07-26T06:18:59.247696+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:02.107634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.257080+0000","last_change":"2024-07-26T06:18:59.251750+0000","last_active":"2024-07-26T06:18:59.257080+0000","last_peered":"2024-07-26T06:18:59.257080+0000","last_clean":"2024-07-26T06:18:59.257080+0000","last_became_active":"2024-07-26T06:18:59.250623+0000","last_became_peered":"2024-07-26T06:18:59.250623+0000","last_unstale":"2024-07-26T06:18:59.257080+0000","last_undegraded":"2024-07-26T06:18:59.257080+0000","last_fullsized":"2024-07-26T06:18:59.257080+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:13.948724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245833+0000","last_change":"2024-07-26T06:18:59.246414+0000","last_active":"2024-07-26T06:19:00.245833+0000","last_peered":"2024-07-26T06:19:00.245833+0000","last_clean":"2024-07-26T06:19:00.245833+0000","last_became_active":"2024-07-26T06:18:59.244670+0000","last_became_peered":"2024-07-26T06:18:59.244670+0000","last_unstale":"2024-07-26T06:19:00.245833+0000","last_undegraded":"2024-07-26T06:19:00.245833+0000","last_fullsized":"2024-07-26T06:19:00.245833+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:04:53.973898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245840+0000","last_change":"2024-07-26T06:18:59.245794+0000","last_active":"2024-07-26T06:19:00.245840+0000","last_peered":"2024-07-26T06:19:00.245840+0000","last_clean":"2024-07-26T06:19:00.245840+0000","last_became_active":"2024-07-26T06:18:59.245303+0000","last_became_peered":"2024-07-26T06:18:59.245303+0000","last_unstale":"2024-07-26T06:19:00.245840+0000","last_undegraded":"2024-07-26T06:19:00.245840+0000","last_fullsized":"2024-07-26T06:19:00.245840+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:40:18.725897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244487+0000","last_change":"2024-07-26T06:18:59.239312+0000","last_active":"2024-07-26T06:19:00.244487+0000","last_peered":"2024-07-26T06:19:00.244487+0000","last_clean":"2024-07-26T06:19:00.244487+0000","last_became_active":"2024-07-26T06:18:59.238852+0000","last_became_peered":"2024-07-26T06:18:59.238852+0000","last_unstale":"2024-07-26T06:19:00.244487+0000","last_undegraded":"2024-07-26T06:19:00.244487+0000","last_fullsized":"2024-07-26T06:19:00.244487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:36:25.099789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244837+0000","last_change":"2024-07-26T06:18:59.251462+0000","last_active":"2024-07-26T06:19:00.244837+0000","last_peered":"2024-07-26T06:19:00.244837+0000","last_clean":"2024-07-26T06:19:00.244837+0000","last_became_active":"2024-07-26T06:18:59.246782+0000","last_became_peered":"2024-07-26T06:18:59.246782+0000","last_unstale":"2024-07-26T06:19:00.244837+0000","last_undegraded":"2024-07-26T06:19:00.244837+0000","last_fullsized":"2024-07-26T06:19:00.244837+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:53.912448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244315+0000","last_change":"2024-07-26T06:18:59.247489+0000","last_active":"2024-07-26T06:19:00.244315+0000","last_peered":"2024-07-26T06:19:00.244315+0000","last_clean":"2024-07-26T06:19:00.244315+0000","last_became_active":"2024-07-26T06:18:59.247009+0000","last_became_peered":"2024-07-26T06:18:59.247009+0000","last_unstale":"2024-07-26T06:19:00.244315+0000","last_undegraded":"2024-07-26T06:19:00.244315+0000","last_fullsized":"2024-07-26T06:19:00.244315+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:40:33.922534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.247002+0000","last_change":"2024-07-26T06:18:59.232772+0000","last_active":"2024-07-26T06:19:00.247002+0000","last_peered":"2024-07-26T06:19:00.247002+0000","last_clean":"2024-07-26T06:19:00.247002+0000","last_became_active":"2024-07-26T06:18:59.232446+0000","last_became_peered":"2024-07-26T06:18:59.232446+0000","last_unstale":"2024-07-26T06:19:00.247002+0000","last_undegraded":"2024-07-26T06:19:00.247002+0000","last_fullsized":"2024-07-26T06:19:00.247002+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:19:45.731961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244765+0000","last_change":"2024-07-26T06:18:59.241701+0000","last_active":"2024-07-26T06:19:00.244765+0000","last_peered":"2024-07-26T06:19:00.244765+0000","last_clean":"2024-07-26T06:19:00.244765+0000","last_became_active":"2024-07-26T06:18:59.239781+0000","last_became_peered":"2024-07-26T06:18:59.239781+0000","last_unstale":"2024-07-26T06:19:00.244765+0000","last_undegraded":"2024-07-26T06:19:00.244765+0000","last_fullsized":"2024-07-26T06:19:00.244765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:41:06.463369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252246+0000","last_change":"2024-07-26T06:18:59.249540+0000","last_active":"2024-07-26T06:18:59.252246+0000","last_peered":"2024-07-26T06:18:59.252246+0000","last_clean":"2024-07-26T06:18:59.252246+0000","last_became_active":"2024-07-26T06:18:59.249195+0000","last_became_peered":"2024-07-26T06:18:59.249195+0000","last_unstale":"2024-07-26T06:18:59.252246+0000","last_undegraded":"2024-07-26T06:18:59.252246+0000","last_fullsized":"2024-07-26T06:18:59.252246+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:02:42.433791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243852+0000","last_change":"2024-07-26T06:18:59.256536+0000","last_active":"2024-07-26T06:19:00.243852+0000","last_peered":"2024-07-26T06:19:00.243852+0000","last_clean":"2024-07-26T06:19:00.243852+0000","last_became_active":"2024-07-26T06:18:59.256275+0000","last_became_peered":"2024-07-26T06:18:59.256275+0000","last_unstale":"2024-07-26T06:19:00.243852+0000","last_undegraded":"2024-07-26T06:19:00.243852+0000","last_fullsized":"2024-07-26T06:19:00.243852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:36:07.775693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244992+0000","last_change":"2024-07-26T06:18:59.245583+0000","last_active":"2024-07-26T06:19:00.244992+0000","last_peered":"2024-07-26T06:19:00.244992+0000","last_clean":"2024-07-26T06:19:00.244992+0000","last_became_active":"2024-07-26T06:18:59.244895+0000","last_became_peered":"2024-07-26T06:18:59.244895+0000","last_unstale":"2024-07-26T06:19:00.244992+0000","last_undegraded":"2024-07-26T06:19:00.244992+0000","last_fullsized":"2024-07-26T06:19:00.244992+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:11:32.476137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.253237+0000","last_change":"2024-07-26T06:18:59.250553+0000","last_active":"2024-07-26T06:18:59.253237+0000","last_peered":"2024-07-26T06:18:59.253237+0000","last_clean":"2024-07-26T06:18:59.253237+0000","last_became_active":"2024-07-26T06:18:59.248806+0000","last_became_peered":"2024-07-26T06:18:59.248806+0000","last_unstale":"2024-07-26T06:18:59.253237+0000","last_undegraded":"2024-07-26T06:18:59.253237+0000","last_fullsized":"2024-07-26T06:18:59.253237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:47:48.276723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242985+0000","last_change":"2024-07-26T06:18:59.240660+0000","last_active":"2024-07-26T06:19:00.242985+0000","last_peered":"2024-07-26T06:19:00.242985+0000","last_clean":"2024-07-26T06:19:00.242985+0000","last_became_active":"2024-07-26T06:18:59.240399+0000","last_became_peered":"2024-07-26T06:18:59.240399+0000","last_unstale":"2024-07-26T06:19:00.242985+0000","last_undegraded":"2024-07-26T06:19:00.242985+0000","last_fullsized":"2024-07-26T06:19:00.242985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:34:56.315636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251952+0000","last_change":"2024-07-26T06:18:59.247410+0000","last_active":"2024-07-26T06:18:59.251952+0000","last_peered":"2024-07-26T06:18:59.251952+0000","last_clean":"2024-07-26T06:18:59.251952+0000","last_became_active":"2024-07-26T06:18:59.247139+0000","last_became_peered":"2024-07-26T06:18:59.247139+0000","last_unstale":"2024-07-26T06:18:59.251952+0000","last_undegraded":"2024-07-26T06:18:59.251952+0000","last_fullsized":"2024-07-26T06:18:59.251952+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:37.273017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.236427+0000","last_change":"2024-07-26T06:18:59.229828+0000","last_active":"2024-07-26T06:18:59.236427+0000","last_peered":"2024-07-26T06:18:59.236427+0000","last_clean":"2024-07-26T06:18:59.236427+0000","last_became_active":"2024-07-26T06:18:59.229558+0000","last_became_peered":"2024-07-26T06:18:59.229558+0000","last_unstale":"2024-07-26T06:18:59.236427+0000","last_undegraded":"2024-07-26T06:18:59.236427+0000","last_fullsized":"2024-07-26T06:18:59.236427+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:27:22.825430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"98'4","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.247505+0000","last_change":"2024-07-26T06:18:58.235125+0000","last_active":"2024-07-26T06:18:59.247505+0000","last_peered":"2024-07-26T06:18:59.247505+0000","last_clean":"2024-07-26T06:18:59.247505+0000","last_became_active":"2024-07-26T06:18:58.234837+0000","last_became_peered":"2024-07-26T06:18:58.234837+0000","last_unstale":"2024-07-26T06:18:59.247505+0000","last_undegraded":"2024-07-26T06:18:59.247505+0000","last_fullsized":"2024-07-26T06:18:59.247505+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:19.852181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.250028+0000","last_change":"2024-07-26T06:18:59.246491+0000","last_active":"2024-07-26T06:18:59.250028+0000","last_peered":"2024-07-26T06:18:59.250028+0000","last_clean":"2024-07-26T06:18:59.250028+0000","last_became_active":"2024-07-26T06:18:59.246199+0000","last_became_peered":"2024-07-26T06:18:59.246199+0000","last_unstale":"2024-07-26T06:18:59.250028+0000","last_undegraded":"2024-07-26T06:18:59.250028+0000","last_fullsized":"2024-07-26T06:18:59.250028+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:18:54.678508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251072+0000","last_change":"2024-07-26T06:18:58.234573+0000","last_active":"2024-07-26T06:18:59.251072+0000","last_peered":"2024-07-26T06:18:59.251072+0000","last_clean":"2024-07-26T06:18:59.251072+0000","last_became_active":"2024-07-26T06:18:58.234260+0000","last_became_peered":"2024-07-26T06:18:58.234260+0000","last_unstale":"2024-07-26T06:18:59.251072+0000","last_undegraded":"2024-07-26T06:18:59.251072+0000","last_fullsized":"2024-07-26T06:18:59.251072+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:47:47.261472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.251838+0000","last_change":"2024-07-26T06:18:59.243764+0000","last_active":"2024-07-26T06:18:59.251838+0000","last_peered":"2024-07-26T06:18:59.251838+0000","last_clean":"2024-07-26T06:18:59.251838+0000","last_became_active":"2024-07-26T06:18:59.243486+0000","last_became_peered":"2024-07-26T06:18:59.243486+0000","last_unstale":"2024-07-26T06:18:59.251838+0000","last_undegraded":"2024-07-26T06:18:59.251838+0000","last_fullsized":"2024-07-26T06:18:59.251838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:01:24.160773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.241918+0000","last_change":"2024-07-26T06:18:58.230586+0000","last_active":"2024-07-26T06:18:59.241918+0000","last_peered":"2024-07-26T06:18:59.241918+0000","last_clean":"2024-07-26T06:18:59.241918+0000","last_became_active":"2024-07-26T06:18:58.230288+0000","last_became_peered":"2024-07-26T06:18:58.230288+0000","last_unstale":"2024-07-26T06:18:59.241918+0000","last_undegraded":"2024-07-26T06:18:59.241918+0000","last_fullsized":"2024-07-26T06:18:59.241918+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:36:47.294190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255826+0000","last_change":"2024-07-26T06:18:59.254060+0000","last_active":"2024-07-26T06:18:59.255826+0000","last_peered":"2024-07-26T06:18:59.255826+0000","last_clean":"2024-07-26T06:18:59.255826+0000","last_became_active":"2024-07-26T06:18:59.251052+0000","last_became_peered":"2024-07-26T06:18:59.251052+0000","last_unstale":"2024-07-26T06:18:59.255826+0000","last_undegraded":"2024-07-26T06:18:59.255826+0000","last_fullsized":"2024-07-26T06:18:59.255826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:32:42.090016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.239240+0000","last_change":"2024-07-26T06:18:58.232832+0000","last_active":"2024-07-26T06:18:59.239240+0000","last_peered":"2024-07-26T06:18:59.239240+0000","last_clean":"2024-07-26T06:18:59.239240+0000","last_became_active":"2024-07-26T06:18:58.228727+0000","last_became_peered":"2024-07-26T06:18:58.228727+0000","last_unstale":"2024-07-26T06:18:59.239240+0000","last_undegraded":"2024-07-26T06:18:59.239240+0000","last_fullsized":"2024-07-26T06:18:59.239240+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:20:49.955403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243468+0000","last_change":"2024-07-26T06:18:59.248460+0000","last_active":"2024-07-26T06:19:00.243468+0000","last_peered":"2024-07-26T06:19:00.243468+0000","last_clean":"2024-07-26T06:19:00.243468+0000","last_became_active":"2024-07-26T06:18:59.248205+0000","last_became_peered":"2024-07-26T06:18:59.248205+0000","last_unstale":"2024-07-26T06:19:00.243468+0000","last_undegraded":"2024-07-26T06:19:00.243468+0000","last_fullsized":"2024-07-26T06:19:00.243468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:20:12.078801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.243034+0000","last_change":"2024-07-26T06:18:58.242210+0000","last_active":"2024-07-26T06:18:59.243034+0000","last_peered":"2024-07-26T06:18:59.243034+0000","last_clean":"2024-07-26T06:18:59.243034+0000","last_became_active":"2024-07-26T06:18:58.239481+0000","last_became_peered":"2024-07-26T06:18:58.239481+0000","last_unstale":"2024-07-26T06:18:59.243034+0000","last_undegraded":"2024-07-26T06:18:59.243034+0000","last_fullsized":"2024-07-26T06:18:59.243034+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.137519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245226+0000","last_change":"2024-07-26T06:18:59.232001+0000","last_active":"2024-07-26T06:19:00.245226+0000","last_peered":"2024-07-26T06:19:00.245226+0000","last_clean":"2024-07-26T06:19:00.245226+0000","last_became_active":"2024-07-26T06:18:59.231407+0000","last_became_peered":"2024-07-26T06:18:59.231407+0000","last_unstale":"2024-07-26T06:19:00.245226+0000","last_undegraded":"2024-07-26T06:19:00.245226+0000","last_fullsized":"2024-07-26T06:19:00.245226+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:43:14.886460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244598+0000","last_change":"2024-07-26T06:18:58.234670+0000","last_active":"2024-07-26T06:19:00.244598+0000","last_peered":"2024-07-26T06:19:00.244598+0000","last_clean":"2024-07-26T06:19:00.244598+0000","last_became_active":"2024-07-26T06:18:58.234400+0000","last_became_peered":"2024-07-26T06:18:58.234400+0000","last_unstale":"2024-07-26T06:19:00.244598+0000","last_undegraded":"2024-07-26T06:19:00.244598+0000","last_fullsized":"2024-07-26T06:19:00.244598+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:16.824909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243192+0000","last_change":"2024-07-26T06:18:59.230966+0000","last_active":"2024-07-26T06:19:00.243192+0000","last_peered":"2024-07-26T06:19:00.243192+0000","last_clean":"2024-07-26T06:19:00.243192+0000","last_became_active":"2024-07-26T06:18:59.230675+0000","last_became_peered":"2024-07-26T06:18:59.230675+0000","last_unstale":"2024-07-26T06:19:00.243192+0000","last_undegraded":"2024-07-26T06:19:00.243192+0000","last_fullsized":"2024-07-26T06:19:00.243192+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:31:58.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244827+0000","last_change":"2024-07-26T06:18:58.236666+0000","last_active":"2024-07-26T06:19:00.244827+0000","last_peered":"2024-07-26T06:19:00.244827+0000","last_clean":"2024-07-26T06:19:00.244827+0000","last_became_active":"2024-07-26T06:18:58.236310+0000","last_became_peered":"2024-07-26T06:18:58.236310+0000","last_unstale":"2024-07-26T06:19:00.244827+0000","last_undegraded":"2024-07-26T06:19:00.244827+0000","last_fullsized":"2024-07-26T06:19:00.244827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:08:25.599277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.253220+0000","last_change":"2024-07-26T06:18:59.248641+0000","last_active":"2024-07-26T06:18:59.253220+0000","last_peered":"2024-07-26T06:18:59.253220+0000","last_clean":"2024-07-26T06:18:59.253220+0000","last_became_active":"2024-07-26T06:18:59.248363+0000","last_became_peered":"2024-07-26T06:18:59.248363+0000","last_unstale":"2024-07-26T06:18:59.253220+0000","last_undegraded":"2024-07-26T06:18:59.253220+0000","last_fullsized":"2024-07-26T06:18:59.253220+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:50.467113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242901+0000","last_change":"2024-07-26T06:18:58.240064+0000","last_active":"2024-07-26T06:19:00.242901+0000","last_peered":"2024-07-26T06:19:00.242901+0000","last_clean":"2024-07-26T06:19:00.242901+0000","last_became_active":"2024-07-26T06:18:58.239535+0000","last_became_peered":"2024-07-26T06:18:58.239535+0000","last_unstale":"2024-07-26T06:19:00.242901+0000","last_undegraded":"2024-07-26T06:19:00.242901+0000","last_fullsized":"2024-07-26T06:19:00.242901+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:07.330988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.245535+0000","last_change":"2024-07-26T06:18:59.242992+0000","last_active":"2024-07-26T06:18:59.245535+0000","last_peered":"2024-07-26T06:18:59.245535+0000","last_clean":"2024-07-26T06:18:59.245535+0000","last_became_active":"2024-07-26T06:18:59.242687+0000","last_became_peered":"2024-07-26T06:18:59.242687+0000","last_unstale":"2024-07-26T06:18:59.245535+0000","last_undegraded":"2024-07-26T06:18:59.245535+0000","last_fullsized":"2024-07-26T06:18:59.245535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:58.939174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.222694+0000","last_change":"2024-07-26T06:18:58.229299+0000","last_active":"2024-07-26T06:18:59.222694+0000","last_peered":"2024-07-26T06:18:59.222694+0000","last_clean":"2024-07-26T06:18:59.222694+0000","last_became_active":"2024-07-26T06:18:58.229002+0000","last_became_peered":"2024-07-26T06:18:58.229002+0000","last_unstale":"2024-07-26T06:18:59.222694+0000","last_undegraded":"2024-07-26T06:18:59.222694+0000","last_fullsized":"2024-07-26T06:18:59.222694+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:26:35.242032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.250006+0000","last_change":"2024-07-26T06:18:59.245708+0000","last_active":"2024-07-26T06:18:59.250006+0000","last_peered":"2024-07-26T06:18:59.250006+0000","last_clean":"2024-07-26T06:18:59.250006+0000","last_became_active":"2024-07-26T06:18:59.244452+0000","last_became_peered":"2024-07-26T06:18:59.244452+0000","last_unstale":"2024-07-26T06:18:59.250006+0000","last_undegraded":"2024-07-26T06:18:59.250006+0000","last_fullsized":"2024-07-26T06:18:59.250006+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:02.522586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242861+0000","last_change":"2024-07-26T06:18:58.239190+0000","last_active":"2024-07-26T06:19:00.242861+0000","last_peered":"2024-07-26T06:19:00.242861+0000","last_clean":"2024-07-26T06:19:00.242861+0000","last_became_active":"2024-07-26T06:18:58.238927+0000","last_became_peered":"2024-07-26T06:18:58.238927+0000","last_unstale":"2024-07-26T06:19:00.242861+0000","last_undegraded":"2024-07-26T06:19:00.242861+0000","last_fullsized":"2024-07-26T06:19:00.242861+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:43:13.871209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244572+0000","last_change":"2024-07-26T06:18:59.248455+0000","last_active":"2024-07-26T06:19:00.244572+0000","last_peered":"2024-07-26T06:19:00.244572+0000","last_clean":"2024-07-26T06:19:00.244572+0000","last_became_active":"2024-07-26T06:18:59.245481+0000","last_became_peered":"2024-07-26T06:18:59.245481+0000","last_unstale":"2024-07-26T06:19:00.244572+0000","last_undegraded":"2024-07-26T06:19:00.244572+0000","last_fullsized":"2024-07-26T06:19:00.244572+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:43.755259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"98'3","reported_seq":17,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244061+0000","last_change":"2024-07-26T06:18:58.240696+0000","last_active":"2024-07-26T06:19:00.244061+0000","last_peered":"2024-07-26T06:19:00.244061+0000","last_clean":"2024-07-26T06:19:00.244061+0000","last_became_active":"2024-07-26T06:18:58.240340+0000","last_became_peered":"2024-07-26T06:18:58.240340+0000","last_unstale":"2024-07-26T06:19:00.244061+0000","last_undegraded":"2024-07-26T06:19:00.244061+0000","last_fullsized":"2024-07-26T06:19:00.244061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:00:50.871302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252545+0000","last_change":"2024-07-26T06:18:59.247262+0000","last_active":"2024-07-26T06:18:59.252545+0000","last_peered":"2024-07-26T06:18:59.252545+0000","last_clean":"2024-07-26T06:18:59.252545+0000","last_became_active":"2024-07-26T06:18:59.246993+0000","last_became_peered":"2024-07-26T06:18:59.246993+0000","last_unstale":"2024-07-26T06:18:59.252545+0000","last_undegraded":"2024-07-26T06:18:59.252545+0000","last_fullsized":"2024-07-26T06:18:59.252545+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:55:44.943255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"98'6","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244515+0000","last_change":"2024-07-26T06:18:58.241636+0000","last_active":"2024-07-26T06:19:00.244515+0000","last_peered":"2024-07-26T06:19:00.244515+0000","last_clean":"2024-07-26T06:19:00.244515+0000","last_became_active":"2024-07-26T06:18:58.240783+0000","last_became_peered":"2024-07-26T06:18:58.240783+0000","last_unstale":"2024-07-26T06:19:00.244515+0000","last_undegraded":"2024-07-26T06:19:00.244515+0000","last_fullsized":"2024-07-26T06:19:00.244515+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:00:29.173157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244538+0000","last_change":"2024-07-26T06:18:59.246256+0000","last_active":"2024-07-26T06:19:00.244538+0000","last_peered":"2024-07-26T06:19:00.244538+0000","last_clean":"2024-07-26T06:19:00.244538+0000","last_became_active":"2024-07-26T06:18:59.245959+0000","last_became_peered":"2024-07-26T06:18:59.245959+0000","last_unstale":"2024-07-26T06:19:00.244538+0000","last_undegraded":"2024-07-26T06:19:00.244538+0000","last_fullsized":"2024-07-26T06:19:00.244538+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:39.726779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.246878+0000","last_change":"2024-07-26T06:18:58.238822+0000","last_active":"2024-07-26T06:18:59.246878+0000","last_peered":"2024-07-26T06:18:59.246878+0000","last_clean":"2024-07-26T06:18:59.246878+0000","last_became_active":"2024-07-26T06:18:58.238445+0000","last_became_peered":"2024-07-26T06:18:58.238445+0000","last_unstale":"2024-07-26T06:18:59.246878+0000","last_undegraded":"2024-07-26T06:18:59.246878+0000","last_fullsized":"2024-07-26T06:18:59.246878+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:19:36.952198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243408+0000","last_change":"2024-07-26T06:18:59.245625+0000","last_active":"2024-07-26T06:19:00.243408+0000","last_peered":"2024-07-26T06:19:00.243408+0000","last_clean":"2024-07-26T06:19:00.243408+0000","last_became_active":"2024-07-26T06:18:59.245367+0000","last_became_peered":"2024-07-26T06:18:59.245367+0000","last_unstale":"2024-07-26T06:19:00.243408+0000","last_undegraded":"2024-07-26T06:19:00.243408+0000","last_fullsized":"2024-07-26T06:19:00.243408+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:46:41.958089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.220982+0000","last_change":"2024-07-26T06:18:58.240538+0000","last_active":"2024-07-26T06:18:59.220982+0000","last_peered":"2024-07-26T06:18:59.220982+0000","last_clean":"2024-07-26T06:18:59.220982+0000","last_became_active":"2024-07-26T06:18:58.240140+0000","last_became_peered":"2024-07-26T06:18:58.240140+0000","last_unstale":"2024-07-26T06:18:59.220982+0000","last_undegraded":"2024-07-26T06:18:59.220982+0000","last_fullsized":"2024-07-26T06:18:59.220982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:24:28.841009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242822+0000","last_change":"2024-07-26T06:18:59.237309+0000","last_active":"2024-07-26T06:19:00.242822+0000","last_peered":"2024-07-26T06:19:00.242822+0000","last_clean":"2024-07-26T06:19:00.242822+0000","last_became_active":"2024-07-26T06:18:59.236769+0000","last_became_peered":"2024-07-26T06:18:59.236769+0000","last_unstale":"2024-07-26T06:19:00.242822+0000","last_undegraded":"2024-07-26T06:19:00.242822+0000","last_fullsized":"2024-07-26T06:19:00.242822+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:38.135036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"98'8","reported_seq":20,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.255125+0000","last_change":"2024-07-26T06:18:58.226227+0000","last_active":"2024-07-26T06:18:59.255125+0000","last_peered":"2024-07-26T06:18:59.255125+0000","last_clean":"2024-07-26T06:18:59.255125+0000","last_became_active":"2024-07-26T06:18:58.225901+0000","last_became_peered":"2024-07-26T06:18:58.225901+0000","last_unstale":"2024-07-26T06:18:59.255125+0000","last_undegraded":"2024-07-26T06:18:59.255125+0000","last_fullsized":"2024-07-26T06:18:59.255125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:28.175721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249756+0000","last_change":"2024-07-26T06:18:59.240235+0000","last_active":"2024-07-26T06:18:59.249756+0000","last_peered":"2024-07-26T06:18:59.249756+0000","last_clean":"2024-07-26T06:18:59.249756+0000","last_became_active":"2024-07-26T06:18:59.239350+0000","last_became_peered":"2024-07-26T06:18:59.239350+0000","last_unstale":"2024-07-26T06:18:59.249756+0000","last_undegraded":"2024-07-26T06:18:59.249756+0000","last_fullsized":"2024-07-26T06:18:59.249756+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.098824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"98'1","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245497+0000","last_change":"2024-07-26T06:18:58.235735+0000","last_active":"2024-07-26T06:19:00.245497+0000","last_peered":"2024-07-26T06:19:00.245497+0000","last_clean":"2024-07-26T06:19:00.245497+0000","last_became_active":"2024-07-26T06:18:58.235305+0000","last_became_peered":"2024-07-26T06:18:58.235305+0000","last_unstale":"2024-07-26T06:19:00.245497+0000","last_undegraded":"2024-07-26T06:19:00.245497+0000","last_fullsized":"2024-07-26T06:19:00.245497+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:41:35.858030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249664+0000","last_change":"2024-07-26T06:18:59.242674+0000","last_active":"2024-07-26T06:18:59.249664+0000","last_peered":"2024-07-26T06:18:59.249664+0000","last_clean":"2024-07-26T06:18:59.249664+0000","last_became_active":"2024-07-26T06:18:59.242280+0000","last_became_peered":"2024-07-26T06:18:59.242280+0000","last_unstale":"2024-07-26T06:18:59.249664+0000","last_undegraded":"2024-07-26T06:18:59.249664+0000","last_fullsized":"2024-07-26T06:18:59.249664+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:54:19.159552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249700+0000","last_change":"2024-07-26T06:18:58.230026+0000","last_active":"2024-07-26T06:18:59.249700+0000","last_peered":"2024-07-26T06:18:59.249700+0000","last_clean":"2024-07-26T06:18:59.249700+0000","last_became_active":"2024-07-26T06:18:58.229730+0000","last_became_peered":"2024-07-26T06:18:58.229730+0000","last_unstale":"2024-07-26T06:18:59.249700+0000","last_undegraded":"2024-07-26T06:18:59.249700+0000","last_fullsized":"2024-07-26T06:18:59.249700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:32:49.133605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.260650+0000","last_change":"2024-07-26T06:18:59.257176+0000","last_active":"2024-07-26T06:18:59.260650+0000","last_peered":"2024-07-26T06:18:59.260650+0000","last_clean":"2024-07-26T06:18:59.260650+0000","last_became_active":"2024-07-26T06:18:59.256831+0000","last_became_peered":"2024-07-26T06:18:59.256831+0000","last_unstale":"2024-07-26T06:18:59.260650+0000","last_undegraded":"2024-07-26T06:18:59.260650+0000","last_fullsized":"2024-07-26T06:18:59.260650+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:02.107634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.246099+0000","last_change":"2024-07-26T06:18:58.241191+0000","last_active":"2024-07-26T06:18:59.246099+0000","last_peered":"2024-07-26T06:18:59.246099+0000","last_clean":"2024-07-26T06:18:59.246099+0000","last_became_active":"2024-07-26T06:18:58.241027+0000","last_became_peered":"2024-07-26T06:18:58.241027+0000","last_unstale":"2024-07-26T06:18:59.246099+0000","last_undegraded":"2024-07-26T06:18:59.246099+0000","last_fullsized":"2024-07-26T06:18:59.246099+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:12.933473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.256334+0000","last_change":"2024-07-26T06:18:59.254766+0000","last_active":"2024-07-26T06:18:59.256334+0000","last_peered":"2024-07-26T06:18:59.256334+0000","last_clean":"2024-07-26T06:18:59.256334+0000","last_became_active":"2024-07-26T06:18:59.254460+0000","last_became_peered":"2024-07-26T06:18:59.254460+0000","last_unstale":"2024-07-26T06:18:59.256334+0000","last_undegraded":"2024-07-26T06:18:59.256334+0000","last_fullsized":"2024-07-26T06:18:59.256334+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:47:11.658319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.233594+0000","last_change":"2024-07-26T06:18:58.240084+0000","last_active":"2024-07-26T06:18:59.233594+0000","last_peered":"2024-07-26T06:18:59.233594+0000","last_clean":"2024-07-26T06:18:59.233594+0000","last_became_active":"2024-07-26T06:18:58.239739+0000","last_became_peered":"2024-07-26T06:18:58.239739+0000","last_unstale":"2024-07-26T06:18:59.233594+0000","last_undegraded":"2024-07-26T06:18:59.233594+0000","last_fullsized":"2024-07-26T06:18:59.233594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:59.327981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.252826+0000","last_change":"2024-07-26T06:18:59.247942+0000","last_active":"2024-07-26T06:18:59.252826+0000","last_peered":"2024-07-26T06:18:59.252826+0000","last_clean":"2024-07-26T06:18:59.252826+0000","last_became_active":"2024-07-26T06:18:59.247669+0000","last_became_peered":"2024-07-26T06:18:59.247669+0000","last_unstale":"2024-07-26T06:18:59.252826+0000","last_undegraded":"2024-07-26T06:18:59.252826+0000","last_fullsized":"2024-07-26T06:18:59.252826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:35.380377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.239900+0000","last_change":"2024-07-26T06:18:58.236979+0000","last_active":"2024-07-26T06:18:59.239900+0000","last_peered":"2024-07-26T06:18:59.239900+0000","last_clean":"2024-07-26T06:18:59.239900+0000","last_became_active":"2024-07-26T06:18:58.236704+0000","last_became_peered":"2024-07-26T06:18:58.236704+0000","last_unstale":"2024-07-26T06:18:59.239900+0000","last_undegraded":"2024-07-26T06:18:59.239900+0000","last_fullsized":"2024-07-26T06:18:59.239900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:42:10.751849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243941+0000","last_change":"2024-07-26T06:18:59.244713+0000","last_active":"2024-07-26T06:19:00.243941+0000","last_peered":"2024-07-26T06:19:00.243941+0000","last_clean":"2024-07-26T06:19:00.243941+0000","last_became_active":"2024-07-26T06:18:59.244331+0000","last_became_peered":"2024-07-26T06:18:59.244331+0000","last_unstale":"2024-07-26T06:19:00.243941+0000","last_undegraded":"2024-07-26T06:19:00.243941+0000","last_fullsized":"2024-07-26T06:19:00.243941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:11:40.535982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.222583+0000","last_change":"2024-07-26T06:18:58.232007+0000","last_active":"2024-07-26T06:18:59.222583+0000","last_peered":"2024-07-26T06:18:59.222583+0000","last_clean":"2024-07-26T06:18:59.222583+0000","last_became_active":"2024-07-26T06:18:58.231746+0000","last_became_peered":"2024-07-26T06:18:58.231746+0000","last_unstale":"2024-07-26T06:18:59.222583+0000","last_undegraded":"2024-07-26T06:18:59.222583+0000","last_fullsized":"2024-07-26T06:18:59.222583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:10:05.393787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.254132+0000","last_change":"2024-07-26T06:18:59.252085+0000","last_active":"2024-07-26T06:18:59.254132+0000","last_peered":"2024-07-26T06:18:59.254132+0000","last_clean":"2024-07-26T06:18:59.254132+0000","last_became_active":"2024-07-26T06:18:59.250358+0000","last_became_peered":"2024-07-26T06:18:59.250358+0000","last_unstale":"2024-07-26T06:18:59.254132+0000","last_undegraded":"2024-07-26T06:18:59.254132+0000","last_fullsized":"2024-07-26T06:18:59.254132+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:06.334942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.222035+0000","last_change":"2024-07-26T06:18:58.232662+0000","last_active":"2024-07-26T06:18:59.222035+0000","last_peered":"2024-07-26T06:18:59.222035+0000","last_clean":"2024-07-26T06:18:59.222035+0000","last_became_active":"2024-07-26T06:18:58.232338+0000","last_became_peered":"2024-07-26T06:18:58.232338+0000","last_unstale":"2024-07-26T06:18:59.222035+0000","last_undegraded":"2024-07-26T06:18:59.222035+0000","last_fullsized":"2024-07-26T06:18:59.222035+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:03:41.924302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.242985+0000","last_change":"2024-07-26T06:18:59.257900+0000","last_active":"2024-07-26T06:19:00.242985+0000","last_peered":"2024-07-26T06:19:00.242985+0000","last_clean":"2024-07-26T06:19:00.242985+0000","last_became_active":"2024-07-26T06:18:59.257599+0000","last_became_peered":"2024-07-26T06:18:59.257599+0000","last_unstale":"2024-07-26T06:19:00.242985+0000","last_undegraded":"2024-07-26T06:19:00.242985+0000","last_fullsized":"2024-07-26T06:19:00.242985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:20:03.140222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243046+0000","last_change":"2024-07-26T06:18:58.238882+0000","last_active":"2024-07-26T06:19:00.243046+0000","last_peered":"2024-07-26T06:19:00.243046+0000","last_clean":"2024-07-26T06:19:00.243046+0000","last_became_active":"2024-07-26T06:18:58.235910+0000","last_became_peered":"2024-07-26T06:18:58.235910+0000","last_unstale":"2024-07-26T06:19:00.243046+0000","last_undegraded":"2024-07-26T06:19:00.243046+0000","last_fullsized":"2024-07-26T06:19:00.243046+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:17:19.617245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.249198+0000","last_change":"2024-07-26T06:18:59.244015+0000","last_active":"2024-07-26T06:18:59.249198+0000","last_peered":"2024-07-26T06:18:59.249198+0000","last_clean":"2024-07-26T06:18:59.249198+0000","last_became_active":"2024-07-26T06:18:59.243739+0000","last_became_peered":"2024-07-26T06:18:59.243739+0000","last_unstale":"2024-07-26T06:18:59.249198+0000","last_undegraded":"2024-07-26T06:18:59.249198+0000","last_fullsized":"2024-07-26T06:18:59.249198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:05:42.409469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"98'2","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243270+0000","last_change":"2024-07-26T06:18:58.240969+0000","last_active":"2024-07-26T06:19:00.243270+0000","last_peered":"2024-07-26T06:19:00.243270+0000","last_clean":"2024-07-26T06:19:00.243270+0000","last_became_active":"2024-07-26T06:18:58.240652+0000","last_became_peered":"2024-07-26T06:18:58.240652+0000","last_unstale":"2024-07-26T06:19:00.243270+0000","last_undegraded":"2024-07-26T06:19:00.243270+0000","last_fullsized":"2024-07-26T06:19:00.243270+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:51.735453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245533+0000","last_change":"2024-07-26T06:18:59.236493+0000","last_active":"2024-07-26T06:19:00.245533+0000","last_peered":"2024-07-26T06:19:00.245533+0000","last_clean":"2024-07-26T06:19:00.245533+0000","last_became_active":"2024-07-26T06:18:59.233060+0000","last_became_peered":"2024-07-26T06:18:59.233060+0000","last_unstale":"2024-07-26T06:19:00.245533+0000","last_undegraded":"2024-07-26T06:19:00.245533+0000","last_fullsized":"2024-07-26T06:19:00.245533+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:22:38.466217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.227386+0000","last_change":"2024-07-26T06:18:58.233524+0000","last_active":"2024-07-26T06:18:59.227386+0000","last_peered":"2024-07-26T06:18:59.227386+0000","last_clean":"2024-07-26T06:18:59.227386+0000","last_became_active":"2024-07-26T06:18:58.233181+0000","last_became_peered":"2024-07-26T06:18:58.233181+0000","last_unstale":"2024-07-26T06:18:59.227386+0000","last_undegraded":"2024-07-26T06:18:59.227386+0000","last_fullsized":"2024-07-26T06:18:59.227386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:55:32.703489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244374+0000","last_change":"2024-07-26T06:18:59.230457+0000","last_active":"2024-07-26T06:19:00.244374+0000","last_peered":"2024-07-26T06:19:00.244374+0000","last_clean":"2024-07-26T06:19:00.244374+0000","last_became_active":"2024-07-26T06:18:59.224988+0000","last_became_peered":"2024-07-26T06:18:59.224988+0000","last_unstale":"2024-07-26T06:19:00.244374+0000","last_undegraded":"2024-07-26T06:19:00.244374+0000","last_fullsized":"2024-07-26T06:19:00.244374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:06:56.302690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244194+0000","last_change":"2024-07-26T06:18:58.235818+0000","last_active":"2024-07-26T06:19:00.244194+0000","last_peered":"2024-07-26T06:19:00.244194+0000","last_clean":"2024-07-26T06:19:00.244194+0000","last_became_active":"2024-07-26T06:18:58.235451+0000","last_became_peered":"2024-07-26T06:18:58.235451+0000","last_unstale":"2024-07-26T06:19:00.244194+0000","last_undegraded":"2024-07-26T06:19:00.244194+0000","last_fullsized":"2024-07-26T06:19:00.244194+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:16.797214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.257047+0000","last_change":"2024-07-26T06:18:59.255794+0000","last_active":"2024-07-26T06:18:59.257047+0000","last_peered":"2024-07-26T06:18:59.257047+0000","last_clean":"2024-07-26T06:18:59.257047+0000","last_became_active":"2024-07-26T06:18:59.254994+0000","last_became_peered":"2024-07-26T06:18:59.254994+0000","last_unstale":"2024-07-26T06:18:59.257047+0000","last_undegraded":"2024-07-26T06:18:59.257047+0000","last_fullsized":"2024-07-26T06:18:59.257047+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:53:08.346239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244482+0000","last_change":"2024-07-26T06:18:58.238066+0000","last_active":"2024-07-26T06:19:00.244482+0000","last_peered":"2024-07-26T06:19:00.244482+0000","last_clean":"2024-07-26T06:19:00.244482+0000","last_became_active":"2024-07-26T06:18:58.235846+0000","last_became_peered":"2024-07-26T06:18:58.235846+0000","last_unstale":"2024-07-26T06:19:00.244482+0000","last_undegraded":"2024-07-26T06:19:00.244482+0000","last_fullsized":"2024-07-26T06:19:00.244482+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:04:52.958647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245165+0000","last_change":"2024-07-26T06:18:59.239981+0000","last_active":"2024-07-26T06:19:00.245165+0000","last_peered":"2024-07-26T06:19:00.245165+0000","last_clean":"2024-07-26T06:19:00.245165+0000","last_became_active":"2024-07-26T06:18:59.239654+0000","last_became_peered":"2024-07-26T06:18:59.239654+0000","last_unstale":"2024-07-26T06:19:00.245165+0000","last_undegraded":"2024-07-26T06:19:00.245165+0000","last_fullsized":"2024-07-26T06:19:00.245165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:52.750704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.229936+0000","last_change":"2024-07-26T06:18:58.232570+0000","last_active":"2024-07-26T06:18:59.229936+0000","last_peered":"2024-07-26T06:18:59.229936+0000","last_clean":"2024-07-26T06:18:59.229936+0000","last_became_active":"2024-07-26T06:18:58.232239+0000","last_became_peered":"2024-07-26T06:18:58.232239+0000","last_unstale":"2024-07-26T06:18:59.229936+0000","last_undegraded":"2024-07-26T06:18:59.229936+0000","last_fullsized":"2024-07-26T06:18:59.229936+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:41:51.104522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245487+0000","last_change":"2024-07-26T06:18:59.239077+0000","last_active":"2024-07-26T06:19:00.245487+0000","last_peered":"2024-07-26T06:19:00.245487+0000","last_clean":"2024-07-26T06:19:00.245487+0000","last_became_active":"2024-07-26T06:18:59.237189+0000","last_became_peered":"2024-07-26T06:18:59.237189+0000","last_unstale":"2024-07-26T06:19:00.245487+0000","last_undegraded":"2024-07-26T06:19:00.245487+0000","last_fullsized":"2024-07-26T06:19:00.245487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:17.840160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.245481+0000","last_change":"2024-07-26T06:18:58.236663+0000","last_active":"2024-07-26T06:19:00.245481+0000","last_peered":"2024-07-26T06:19:00.245481+0000","last_clean":"2024-07-26T06:19:00.245481+0000","last_became_active":"2024-07-26T06:18:58.236174+0000","last_became_peered":"2024-07-26T06:18:58.236174+0000","last_unstale":"2024-07-26T06:19:00.245481+0000","last_undegraded":"2024-07-26T06:19:00.245481+0000","last_fullsized":"2024-07-26T06:19:00.245481+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:49.134754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.254091+0000","last_change":"2024-07-26T06:18:59.251090+0000","last_active":"2024-07-26T06:18:59.254091+0000","last_peered":"2024-07-26T06:18:59.254091+0000","last_clean":"2024-07-26T06:18:59.254091+0000","last_became_active":"2024-07-26T06:18:59.250766+0000","last_became_peered":"2024-07-26T06:18:59.250766+0000","last_unstale":"2024-07-26T06:18:59.254091+0000","last_undegraded":"2024-07-26T06:18:59.254091+0000","last_fullsized":"2024-07-26T06:18:59.254091+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:10:24.921556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.223661+0000","last_change":"2024-07-26T06:18:58.231335+0000","last_active":"2024-07-26T06:18:59.223661+0000","last_peered":"2024-07-26T06:18:59.223661+0000","last_clean":"2024-07-26T06:18:59.223661+0000","last_became_active":"2024-07-26T06:18:58.227857+0000","last_became_peered":"2024-07-26T06:18:58.227857+0000","last_unstale":"2024-07-26T06:18:59.223661+0000","last_undegraded":"2024-07-26T06:18:59.223661+0000","last_fullsized":"2024-07-26T06:18:59.223661+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:11:00.503756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243161+0000","last_change":"2024-07-26T06:18:59.232505+0000","last_active":"2024-07-26T06:19:00.243161+0000","last_peered":"2024-07-26T06:19:00.243161+0000","last_clean":"2024-07-26T06:19:00.243161+0000","last_became_active":"2024-07-26T06:18:59.225519+0000","last_became_peered":"2024-07-26T06:18:59.225519+0000","last_unstale":"2024-07-26T06:19:00.243161+0000","last_undegraded":"2024-07-26T06:19:00.243161+0000","last_fullsized":"2024-07-26T06:19:00.243161+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:05.002802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.243413+0000","last_change":"2024-07-26T06:18:58.239275+0000","last_active":"2024-07-26T06:19:00.243413+0000","last_peered":"2024-07-26T06:19:00.243413+0000","last_clean":"2024-07-26T06:19:00.243413+0000","last_became_active":"2024-07-26T06:18:58.235553+0000","last_became_peered":"2024-07-26T06:18:58.235553+0000","last_unstale":"2024-07-26T06:19:00.243413+0000","last_undegraded":"2024-07-26T06:19:00.243413+0000","last_fullsized":"2024-07-26T06:19:00.243413+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:44:53.015361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-07-26T06:19:00.244090+0000","last_change":"2024-07-26T06:18:59.247263+0000","last_active":"2024-07-26T06:19:00.244090+0000","last_peered":"2024-07-26T06:19:00.244090+0000","last_clean":"2024-07-26T06:19:00.244090+0000","last_became_active":"2024-07-26T06:18:59.246666+0000","last_became_peered":"2024-07-26T06:18:59.246666+0000","last_unstale":"2024-07-26T06:19:00.244090+0000","last_undegraded":"2024-07-26T06:19:00.244090+0000","last_fullsized":"2024-07-26T06:19:00.244090+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:58.185560+0000","last_clean_scrub_stamp":"2024-07-26T06:18:58.185560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:09:31.569170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.231831+0000","last_change":"2024-07-26T06:18:58.240401+0000","last_active":"2024-07-26T06:18:59.231831+0000","last_peered":"2024-07-26T06:18:59.231831+0000","last_clean":"2024-07-26T06:18:59.231831+0000","last_became_active":"2024-07-26T06:18:58.240053+0000","last_became_peered":"2024-07-26T06:18:58.240053+0000","last_unstale":"2024-07-26T06:18:59.231831+0000","last_undegraded":"2024-07-26T06:18:59.231831+0000","last_fullsized":"2024-07-26T06:18:59.231831+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:18:57.170309+0000","last_clean_scrub_stamp":"2024-07-26T06:18:57.170309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:51.142931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'79","reported_seq":278,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-07-26T06:18:59.238624+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:18:59.238624+0000","last_peered":"2024-07-26T06:18:59.238624+0000","last_clean":"2024-07-26T06:18:59.238624+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:18:59.238624+0000","last_undegraded":"2024-07-26T06:18:59.238624+0000","last_fullsized":"2024-07-26T06:18:59.238624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509511,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32960,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":1814528,"data_stored":1119423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509512,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":528509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1220608,"data_stored":529043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542215,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1212416,"data_stored":528439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1228800,"data_stored":530121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":528439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542216,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1220608,"data_stored":528563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509512,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1120531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:19:04.185 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:19:04.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-07-26T06:19:04.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 -- 172.21.15.163:0/3089843533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 msgr2=0x7f45540ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 --2- 172.21.15.163:0/3089843533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540ce2f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f453c005fd0 tx=0x7f453c02a410 comp rx=0 tx=0).stop 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 -- 172.21.15.163:0/3089843533 shutdown_connections 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 --2- 172.21.15.163:0/3089843533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45540ce830 0x7f45540b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 --2- 172.21.15.163:0/3089843533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540ce2f0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 -- 172.21.15.163:0/3089843533 >> 172.21.15.163:0/3089843533 conn(0x7f45540cfda0 msgr2=0x7f45540d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 -- 172.21.15.163:0/3089843533 shutdown_connections 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.434+0000 7f4553200640 1 -- 172.21.15.163:0/3089843533 wait complete. 2024-07-26T06:19:04.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.436+0000 7f4553200640 1 Processor -- start 2024-07-26T06:19:04.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.436+0000 7f4553200640 1 -- start start 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4553200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45540b1e80 0x7f45541f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4553200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540ce830 0x7f4554152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4553200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4553200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45540b5130 con 0x7f455415d5a0 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4553200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45540b4fb0 con 0x7f45540ce830 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4553200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45540b52b0 con 0x7f45540b1e80 2024-07-26T06:19:04.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4552800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4552800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35938/0 (socket says 172.21.15.163:35938) 2024-07-26T06:19:04.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.437+0000 7f4552800640 1 -- 172.21.15.163:0/4188935187 learned_addr learned my addr 172.21.15.163:0/4188935187 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4551e00640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45540b1e80 0x7f45541f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4551400640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540ce830 0x7f4554152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4552800640 1 -- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45540b1e80 msgr2=0x7f45541f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4552800640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45540b1e80 0x7f45541f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4552800640 1 -- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540ce830 msgr2=0x7f4554152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4552800640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540ce830 0x7f4554152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4552800640 1 -- 172.21.15.163:0/4188935187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45541f7360 con 0x7f455415d5a0 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4552800640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540cc640 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f454c00c780 tx=0x7f454c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c004b60 con 0x7f455415d5a0 2024-07-26T06:19:04.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f454c004df0 con 0x7f455415d5a0 2024-07-26T06:19:04.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c00aa70 con 0x7f455415d5a0 2024-07-26T06:19:04.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.438+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45541f7640 con 0x7f455415d5a0 2024-07-26T06:19:04.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.439+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45541f7c00 con 0x7f455415d5a0 2024-07-26T06:19:04.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.440+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4554074680 con 0x7f455415d5a0 2024-07-26T06:19:04.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.441+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f454c00ac10 con 0x7f455415d5a0 2024-07-26T06:19:04.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.442+0000 7f454a000640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45300763f0 0x7f45300788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.442+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5709+0+0 (secure 0 0 0) 0x7f454c097c20 con 0x7f455415d5a0 2024-07-26T06:19:04.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.442+0000 7f4551e00640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45300763f0 0x7f45300788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.443+0000 7f4551e00640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45300763f0 0x7f45300788b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f453c033000 tx=0x7f453c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:04.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.446+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4554074680 con 0x7f455415d5a0 2024-07-26T06:19:04.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.733+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f45541f7e00 con 0x7f455415d5a0 2024-07-26T06:19:04.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.738+0000 7f454a000640 1 -- 172.21.15.163:0/4188935187 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f45541f7e00 con 0x7f455415d5a0 2024-07-26T06:19:04.740 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-07-26T06:19:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBOPqNmTOkvBhAALVPc4RSu1c+txetuJMYOew== 2024-07-26T06:19:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: caps mds = "allow" 2024-07-26T06:19:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: caps mon = "allow r" 2024-07-26T06:19:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-07-26T06:19:04.740 INFO:teuthology.orchestra.run.smithi163.stderr:updated caps for client.0 2024-07-26T06:19:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45300763f0 msgr2=0x7f45300788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45300763f0 0x7f45300788b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f453c033000 tx=0x7f453c004730 comp rx=0 tx=0).stop 2024-07-26T06:19:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 msgr2=0x7f45540cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540cc640 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f454c00c780 tx=0x7f454c00cc40 comp rx=0 tx=0).stop 2024-07-26T06:19:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 shutdown_connections 2024-07-26T06:19:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f45300763f0 0x7f45300788b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f455415d5a0 0x7f45540cc640 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540ce830 0x7f4554152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 --2- 172.21.15.163:0/4188935187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45540b1e80 0x7f45541f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 >> 172.21.15.163:0/4188935187 conn(0x7f45540cfda0 msgr2=0x7f45540d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.741+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 shutdown_connections 2024-07-26T06:19:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.742+0000 7f4553200640 1 -- 172.21.15.163:0/4188935187 wait complete. 2024-07-26T06:19:04.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:19:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 -- 172.21.15.163:0/1915399627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 msgr2=0x7f1d300cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 --2- 172.21.15.163:0/1915399627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d300cbd40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009830 tx=0x7f1d1c02a510 comp rx=0 tx=0).stop 2024-07-26T06:19:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 -- 172.21.15.163:0/1915399627 shutdown_connections 2024-07-26T06:19:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 --2- 172.21.15.163:0/1915399627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d300cc280 0x7f1d300b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 --2- 172.21.15.163:0/1915399627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d300cbd40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 -- 172.21.15.163:0/1915399627 >> 172.21.15.163:0/1915399627 conn(0x7f1d300b0e30 msgr2=0x7f1d300b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 -- 172.21.15.163:0/1915399627 shutdown_connections 2024-07-26T06:19:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.963+0000 7f1d34a00640 1 -- 172.21.15.163:0/1915399627 wait complete. 2024-07-26T06:19:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.965+0000 7f1d34a00640 1 Processor -- start 2024-07-26T06:19:04.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.965+0000 7f1d34a00640 1 -- start start 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d34a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d30139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d34a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 0x7f1d301440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d34a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d30146be0 0x7f1d3013ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d34a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1d300b4020 con 0x7f1d300cc280 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d34a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1d300b3ea0 con 0x7f1d300adff0 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d34a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1d300b41a0 con 0x7f1d30146be0 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d2ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 0x7f1d301440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d2ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 0x7f1d301440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45660/0 (socket says 172.21.15.163:45660) 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.966+0000 7f1d2ea00640 1 -- 172.21.15.163:0/3678869175 learned_addr learned my addr 172.21.15.163:0/3678869175 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2fe00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d30146be0 0x7f1d3013ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2f400640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d30139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2ea00640 1 -- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d30146be0 msgr2=0x7f1d3013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2ea00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d30146be0 0x7f1d3013ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2ea00640 1 -- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 msgr2=0x7f1d30139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2ea00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d30139a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2ea00640 1 -- 172.21.15.163:0/3678869175 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d30145b00 con 0x7f1d300cc280 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2f400640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d30139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:04.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d2ea00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 0x7f1d301440f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009950 tx=0x7f1d1c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:04.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.967+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c040070 con 0x7f1d300cc280 2024-07-26T06:19:04.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.968+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d1c0038e0 con 0x7f1d300cc280 2024-07-26T06:19:04.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.968+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d30144a60 con 0x7f1d300cc280 2024-07-26T06:19:04.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.968+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c003d30 con 0x7f1d300cc280 2024-07-26T06:19:04.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.968+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d30145020 con 0x7f1d300cc280 2024-07-26T06:19:04.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.969+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d300b4700 con 0x7f1d300cc280 2024-07-26T06:19:04.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.970+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f1d1c039040 con 0x7f1d300cc280 2024-07-26T06:19:04.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.971+0000 7f1d27400640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1d100763f0 0x7f1d100788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:04.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.971+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5767+0+0 (secure 0 0 0) 0x7f1d1c0b9290 con 0x7f1d300cc280 2024-07-26T06:19:04.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.971+0000 7f1d2f400640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1d100763f0 0x7f1d100788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:04.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.972+0000 7f1d2f400640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1d100763f0 0x7f1d100788b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d30142f70 tx=0x7f1d20009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:04.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:04.975+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d1c082870 con 0x7f1d300cc280 2024-07-26T06:19:05.183 INFO:tasks.ceph.mds.c.smithi163.stderr:2024-07-26T06:19:05.182+0000 7f8c08400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:19:05.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.219+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d300b00b0 con 0x7f1d300cc280 2024-07-26T06:19:05.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.221+0000 7f1d27400640 1 -- 172.21.15.163:0/3678869175 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5234 (secure 0 0 0) 0x7f1d1c082170 con 0x7f1d300cc280 2024-07-26T06:19:05.222 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:05.222 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":70,"btime":"2024-07-26T06:19:02:251060+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:58.690202+0000","modified":"2024-07-26T06:19:01.753809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6834/1667488712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1667488712},{"type":"v1","addr":"172.21.15.163:6834","nonce":1667488712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":8}]} 2024-07-26T06:19:05.222 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 70 2024-07-26T06:19:05.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.223+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1d100763f0 msgr2=0x7f1d100788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.223+0000 7f1d34a00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1d100763f0 0x7f1d100788b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d30142f70 tx=0x7f1d20009040 comp rx=0 tx=0).stop 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.223+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 msgr2=0x7f1d301440f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.223+0000 7f1d34a00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 0x7f1d301440f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009950 tx=0x7f1d1c032c10 comp rx=0 tx=0).stop 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 shutdown_connections 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1d100763f0 0x7f1d100788b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d30146be0 0x7f1d3013ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d300cc280 0x7f1d301440f0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 --2- 172.21.15.163:0/3678869175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d300adff0 0x7f1d30139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 >> 172.21.15.163:0/3678869175 conn(0x7f1d300b0e30 msgr2=0x7f1d300ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 shutdown_connections 2024-07-26T06:19:05.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.224+0000 7f1d34a00640 1 -- 172.21.15.163:0/3678869175 wait complete. 2024-07-26T06:19:05.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T06:18:58.690202+0000', 'modified': '2024-07-26T06:19:01.753809+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6864}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6864': {'gid': 6864, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.163:6834/1667488712', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6832', 'nonce': 1667488712}, {'type': 'v1', 'addr': '172.21.15.163:6834', 'nonce': 1667488712}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6864, 'qdb_cluster': [6864]} 2024-07-26T06:19:05.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T06:19:05.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-07-26T06:19:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 -- 172.21.15.163:0/4089491951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 msgr2=0x7fb9540d1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 --2- 172.21.15.163:0/4089491951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb9540d1450 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009830 tx=0x7fb93c02a5a0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 -- 172.21.15.163:0/4089491951 shutdown_connections 2024-07-26T06:19:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 --2- 172.21.15.163:0/4089491951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9540ceb50 0x7fb9540d31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 --2- 172.21.15.163:0/4089491951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb9540d1450 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 -- 172.21.15.163:0/4089491951 >> 172.21.15.163:0/4089491951 conn(0x7fb95409c7e0 msgr2=0x7fb95408ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 -- 172.21.15.163:0/4089491951 shutdown_connections 2024-07-26T06:19:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.865+0000 7fb95a200640 1 -- 172.21.15.163:0/4089491951 wait complete. 2024-07-26T06:19:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.867+0000 7fb95a200640 1 Processor -- start 2024-07-26T06:19:05.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.867+0000 7fb95a200640 1 -- start start 2024-07-26T06:19:05.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb95a200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9540d3720 0x7fb9541f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb95a200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb954160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb95a200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb95415e740 0x7fb95415ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb95a200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9540b4cc0 con 0x7fb95415d6f0 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb95a200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb9540b4b40 con 0x7fb9540d3720 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb95a200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb9540d0380 con 0x7fb95415e740 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb952a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb954160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb952a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb954160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45686/0 (socket says 172.21.15.163:45686) 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb952a00640 1 -- 172.21.15.163:0/2490361376 learned_addr learned my addr 172.21.15.163:0/2490361376 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb953400640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9540d3720 0x7fb9541f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.868+0000 7fb953e00640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb95415e740 0x7fb95415ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb952a00640 1 -- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb95415e740 msgr2=0x7fb95415ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb952a00640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb95415e740 0x7fb95415ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb952a00640 1 -- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9540d3720 msgr2=0x7fb9541f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb952a00640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9540d3720 0x7fb9541f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb952a00640 1 -- 172.21.15.163:0/2490361376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9540b4400 con 0x7fb95415d6f0 2024-07-26T06:19:05.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb952a00640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb954160910 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fb94400ea50 tx=0x7fb94400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:05.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb94b400640 1 -- 172.21.15.163:0/2490361376 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94400cdc0 con 0x7fb95415d6f0 2024-07-26T06:19:05.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb94b400640 1 -- 172.21.15.163:0/2490361376 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb944011560 con 0x7fb95415d6f0 2024-07-26T06:19:05.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9540cb0d0 con 0x7fb95415d6f0 2024-07-26T06:19:05.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.869+0000 7fb94b400640 1 -- 172.21.15.163:0/2490361376 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9440119b0 con 0x7fb95415d6f0 2024-07-26T06:19:05.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.870+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9540cb690 con 0x7fb95415d6f0 2024-07-26T06:19:05.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.873+0000 7fb94b400640 1 -- 172.21.15.163:0/2490361376 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fb944011b50 con 0x7fb95415d6f0 2024-07-26T06:19:05.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.874+0000 7fb94b400640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb9300763f0 0x7fb9300788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.874+0000 7fb94b400640 1 -- 172.21.15.163:0/2490361376 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5767+0+0 (secure 0 0 0) 0x7fb9440986b0 con 0x7fb95415d6f0 2024-07-26T06:19:05.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.874+0000 7fb953400640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb9300763f0 0x7fb9300788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.874+0000 7fb95d05b740 1 Processor -- start 2024-07-26T06:19:05.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.874+0000 7fb95d05b740 1 -- start start 2024-07-26T06:19:05.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb953400640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb9300763f0 0x7fb9300788b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb93c034000 tx=0x7fb93c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:05.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86ea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e9720 0x55bdd86e7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 0x55bdd869de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb95d05b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bdd85ba3f0 con 0x55bdd86a4ca0 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55bdd86155c0 con 0x55bdd86e9720 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.875+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55bdd8604b60 con 0x55bdd86ea3b0 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 0x55bdd869de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 0x55bdd869de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45688/0 (socket says 172.21.15.163:45688) 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 -- 172.21.15.163:0/1434703952 learned_addr learned my addr 172.21.15.163:0/1434703952 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb952a00640 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e9720 0x55bdd86e7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953400640 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 -- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 msgr2=0x55bdd86ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86ea790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 -- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e9720 msgr2=0x55bdd86e7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e9720 0x55bdd86e7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 -- 172.21.15.163:0/1434703952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd8369850 con 0x55bdd86a4ca0 2024-07-26T06:19:05.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.876+0000 7fb953e00640 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 0x55bdd869de20 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001550 tx=0x7fb94c001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7e6b905799cf04 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:05.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb943400640 1 -- 172.21.15.163:0/1434703952 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c00e6e0 con 0x55bdd86a4ca0 2024-07-26T06:19:05.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb943400640 1 -- 172.21.15.163:0/1434703952 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb94c019070 con 0x55bdd86a4ca0 2024-07-26T06:19:05.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb943400640 1 -- 172.21.15.163:0/1434703952 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c00ec30 con 0x55bdd86a4ca0 2024-07-26T06:19:05.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 -- 172.21.15.163:0/1434703952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 msgr2=0x55bdd869de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 0x55bdd869de20 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001550 tx=0x7fb94c001a10 comp rx=0 tx=0).stop 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 -- 172.21.15.163:0/1434703952 shutdown_connections 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86a4ca0 0x55bdd869de20 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e9720 0x55bdd86e7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1434703952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 -- 172.21.15.163:0/1434703952 >> 172.21.15.163:0/1434703952 conn(0x55bdd869fff0 msgr2=0x55bdd86e8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 -- 172.21.15.163:0/1434703952 shutdown_connections 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.877+0000 7fb95d05b740 1 -- 172.21.15.163:0/1434703952 wait complete. 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.878+0000 7fb95d05b740 1 -- start start 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.878+0000 7fb95d05b740 1 Processor -- start 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86ea3b0 0x55bdd86e3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 0x55bdd86e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bdd85ad210 con 0x55bdd86e0380 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8608ba0 con 0x55bdd86ea3b0 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55bdd86e8b30 con 0x55bdd86bf600 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.879+0000 7fb95d05b740 10 client.0 _ll_register_callbacks cb 0x55bdd85ad8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 0x55bdd86e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 0x55bdd86e4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45694/0 (socket says 172.21.15.163:45694) 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 -- 172.21.15.163:0/1275152048 learned_addr learned my addr 172.21.15.163:0/1275152048 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb953e00640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb953400640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86ea3b0 0x55bdd86e3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 -- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86bf600 msgr2=0x55bdd86e5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 -- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86ea3b0 msgr2=0x55bdd86e3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86ea3b0 0x55bdd86e3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 -- 172.21.15.163:0/1275152048 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd83e06b0 con 0x55bdd86e0380 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb952a00640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 0x55bdd86e4ec0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fb94400c700 tx=0x7fb944061bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766d743e3e21574a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.880+0000 7fb937e00640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb937e00640 1 -- 172.21.15.163:0/1275152048 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944094060 con 0x55bdd86e0380 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb937e00640 1 -- 172.21.15.163:0/1275152048 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 5767+0+0 (secure 0 0 0) 0x7fb94408fb10 con 0x55bdd86e0380 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bdd8617510 con 0x55bdd86e0380 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb937e00640 1 -- 172.21.15.163:0/1275152048 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fb94407f050 con 0x55bdd86e0380 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 10 client.6996 fetch_fsmap learned FSMap version 70 2024-07-26T06:19:05.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) -- 0x55bdd86e14d0 con 0x55bdd86e0380 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb937e00640 1 -- 172.21.15.163:0/1275152048 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55bdd8617510 con 0x55bdd86e0380 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb937e00640 1 -- 172.21.15.163:0/1275152048 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 70) ==== 3502+0+0 (secure 0 0 0) 0x7fb944022030 con 0x55bdd86e0380 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 10 client.6996 fetch_fsmap finished waiting for FSMap version 70 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 10 client.6996 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 20 client.6996 populate_metadata read hostname 'smithi163' 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e2310 0x55bdd8706e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 4 client.6996 mds_command: new command op to 6864 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.881+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 --> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdd8612fb0 con 0x55bdd86e2310 2024-07-26T06:19:05.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.882+0000 7fb953400640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e2310 0x55bdd8706e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.882+0000 7fb953400640 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e2310 0x55bdd8706e80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc930c6a61dfba5 server_cookie=d3800054232e7db7 in_seq=0 out_seq=0 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.882+0000 7fb937e00640 10 client.6996 ms_handle_connect on v2:172.21.15.163:6832/1667488712 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.883+0000 7fb937e00640 1 -- 172.21.15.163:0/1275152048 <== mds.0 v2:172.21.15.163:6832/1667488712 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55bdd8612fb0 con 0x55bdd86e2310 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.883+0000 7fb937e00640 10 client.6996 handle_command_reply: tid=0 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.883+0000 7fb95d05b740 1 client.6996 shutdown 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 msgr2=0x55bdd86e4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 0x55bdd86e4ec0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fb94400c700 tx=0x7fb944061bb0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 shutdown_connections 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e2310 0x55bdd8706e80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86e0380 0x55bdd86e4ec0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1275152048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86ea3b0 0x55bdd86e3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb937e00640 0 client.6996 ms_handle_reset on v2:172.21.15.163:6832/1667488712 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 >> 172.21.15.163:0/1275152048 conn(0x55bdd86a4ca0 msgr2=0x55bdd869cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 shutdown_connections 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 1 -- 172.21.15.163:0/1275152048 wait complete. 2024-07-26T06:19:05.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.884+0000 7fb95d05b740 20 client.6996 trim_cache size 0 max 16384 2024-07-26T06:19:05.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.904+0000 7fb95d05b740 1 Processor -- start 2024-07-26T06:19:05.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.904+0000 7fb95d05b740 1 -- start start 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.905+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.905+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e0380 0x55bdd86a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.905+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86becc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.905+0000 7fb95d05b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9541ff7b0 con 0x55bdd86bf600 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.905+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb9540cce10 con 0x55bdd86e0380 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.905+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb9541ff160 con 0x55bdd86ea3b0 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45704/0 (socket says 172.21.15.163:45704) 2024-07-26T06:19:05.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 -- 172.21.15.163:0/3313705877 learned_addr learned my addr 172.21.15.163:0/3313705877 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 -- 172.21.15.163:0/3313705877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 msgr2=0x55bdd86becc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 -- 172.21.15.163:0/3313705877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e0380 msgr2=0x55bdd86a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e0380 0x55bdd86a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 -- 172.21.15.163:0/3313705877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd86e01b0 con 0x55bdd86bf600 2024-07-26T06:19:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb953400640 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 0x55bdd86e5400 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb93c02ab90 tx=0x7fb93c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2e436a91dffef6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:05.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.906+0000 7fb942a00640 1 -- 172.21.15.163:0/3313705877 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c003b30 con 0x55bdd86bf600 2024-07-26T06:19:05.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb942a00640 1 -- 172.21.15.163:0/3313705877 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb93c003dc0 con 0x55bdd86bf600 2024-07-26T06:19:05.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb942a00640 1 -- 172.21.15.163:0/3313705877 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c04e5f0 con 0x55bdd86bf600 2024-07-26T06:19:05.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 -- 172.21.15.163:0/3313705877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 msgr2=0x55bdd86e5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 0x55bdd86e5400 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb93c02ab90 tx=0x7fb93c03c000 comp rx=0 tx=0).stop 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 -- 172.21.15.163:0/3313705877 shutdown_connections 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86ea3b0 0x55bdd86becc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86e0380 0x55bdd86a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 --2- 172.21.15.163:0/3313705877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86bf600 0x55bdd86e5400 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 -- 172.21.15.163:0/3313705877 >> 172.21.15.163:0/3313705877 conn(0x55bdd86a4ca0 msgr2=0x55bdd869d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 -- 172.21.15.163:0/3313705877 shutdown_connections 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.907+0000 7fb95d05b740 1 -- 172.21.15.163:0/3313705877 wait complete. 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.908+0000 7fb95d05b740 1 -- start start 2024-07-26T06:19:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.908+0000 7fb95d05b740 1 Processor -- start 2024-07-26T06:19:05.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86bf600 0x55bdd86bd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86e0380 0x55bdd86ec150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 0x55bdd8766b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bdd86e2190 con 0x55bdd86ea3b0 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55bdd86cc8a0 con 0x55bdd86bf600 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55bdd86e2720 con 0x55bdd86e0380 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.909+0000 7fb95d05b740 10 client.0 _ll_register_callbacks cb 0x7fb94405dcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 0x55bdd8766b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 0x55bdd8766b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45706/0 (socket says 172.21.15.163:45706) 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 -- 172.21.15.163:0/1204572738 learned_addr learned my addr 172.21.15.163:0/1204572738 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb952a00640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86e0380 0x55bdd86ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953400640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86bf600 0x55bdd86bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 -- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86e0380 msgr2=0x55bdd86ec150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86e0380 0x55bdd86ec150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 -- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86bf600 msgr2=0x55bdd86bd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86bf600 0x55bdd86bd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 -- 172.21.15.163:0/1204572738 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd86cc020 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb953e00640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 0x55bdd8766b00 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001eb0 tx=0x7fb94c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e6288e074bcce4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.910+0000 7fb937e00640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb937e00640 1 -- 172.21.15.163:0/1204572738 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c021070 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb937e00640 1 -- 172.21.15.163:0/1204572738 <== mon.0 v2:172.21.15.163:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 5767+0+0 (secure 0 0 0) 0x7fb94c005450 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb93c003dc0 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb937e00640 1 -- 172.21.15.163:0/1204572738 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fb94c02c050 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 10 client.7002 fetch_fsmap learned FSMap version 70 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) -- 0x55bdd86ec720 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb937e00640 1 -- 172.21.15.163:0/1204572738 <== mon.0 v2:172.21.15.163:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb93c003dc0 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb937e00640 1 -- 172.21.15.163:0/1204572738 <== mon.0 v2:172.21.15.163:3300/0 5 ==== fsmap(e 70) ==== 3502+0+0 (secure 0 0 0) 0x7fb94c02f020 con 0x55bdd86ea3b0 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 10 client.7002 fetch_fsmap finished waiting for FSMap version 70 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 10 client.7002 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 20 client.7002 populate_metadata read hostname 'smithi163' 2024-07-26T06:19:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.911+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e31a0 0x55bdd869d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.912+0000 7fb95d05b740 4 client.7002 mds_command: new command op to 6864 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.912+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 --> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] -- command(tid 0: {"prefix": "status"}) -- 0x55bdd86106f0 con 0x55bdd86e31a0 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.912+0000 7fb953400640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e31a0 0x55bdd869d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.912+0000 7fb953400640 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e31a0 0x55bdd869d9a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3830c48432a958bd server_cookie=472a964ddd3a6a7f in_seq=0 out_seq=0 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.912+0000 7fb937e00640 10 client.7002 ms_handle_connect on v2:172.21.15.163:6832/1667488712 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb937e00640 1 -- 172.21.15.163:0/1204572738 <== mds.0 v2:172.21.15.163:6832/1667488712 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55bdd86106f0 con 0x55bdd86e31a0 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb937e00640 10 client.7002 handle_command_reply: tid=0 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 client.7002 shutdown 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 msgr2=0x55bdd8766b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 0x55bdd8766b00 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001eb0 tx=0x7fb94c00ec10 comp rx=0 tx=0).stop 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 shutdown_connections 2024-07-26T06:19:05.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:6832/1667488712,v1:172.21.15.163:6834/1667488712] conn(0x55bdd86e31a0 0x55bdd869d9a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bdd86ea3b0 0x55bdd8766b00 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bdd86e0380 0x55bdd86ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 --2- 172.21.15.163:0/1204572738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bdd86bf600 0x55bdd86bd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb937e00640 0 client.7002 ms_handle_reset on v2:172.21.15.163:6832/1667488712 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 >> 172.21.15.163:0/1204572738 conn(0x55bdd86a4ca0 msgr2=0x55bdd869d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 shutdown_connections 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 1 -- 172.21.15.163:0/1204572738 wait complete. 2024-07-26T06:19:05.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.913+0000 7fb95d05b740 20 client.7002 trim_cache size 0 max 16384 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_fsid": "a1639b17-d73f-4cae-9471-025ba01f9572", 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "whoami": 0, 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 6864, 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "want_state": "up:active", 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "up:active", 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs", 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "rank_uptime": 6.7043778630000004, 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "mdsmap_epoch": 70, 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch": 100, 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_epoch_barrier": 99, 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout: "uptime": 11.655080264 2024-07-26T06:19:05.919 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:05.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb9300763f0 msgr2=0x7fb9300788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb9300763f0 0x7fb9300788b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb93c034000 tx=0x7fb93c004730 comp rx=0 tx=0).stop 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 msgr2=0x7fb954160910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb954160910 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fb94400ea50 tx=0x7fb94400ef10 comp rx=0 tx=0).stop 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 shutdown_connections 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb9300763f0 0x7fb9300788b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb95415e740 0x7fb95415ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb95415d6f0 0x7fb954160910 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 --2- 172.21.15.163:0/2490361376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9540d3720 0x7fb9541f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 >> 172.21.15.163:0/2490361376 conn(0x7fb95409c7e0 msgr2=0x7fb95408ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 shutdown_connections 2024-07-26T06:19:05.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:05.914+0000 7fb95a200640 1 -- 172.21.15.163:0/2490361376 wait complete. 2024-07-26T06:19:05.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9c00640 1 -- 172.21.15.163:0/3402137683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6c8004800 msgr2=0x7fc6c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/3402137683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6c8004800 0x7fc6c8004c80 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009b20 tx=0x7fc6c8005830 comp rx=0 tx=0).stop 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9200640 1 -- 172.21.15.163:0/3402137683 reap_dead start 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9c00640 1 -- 172.21.15.163:0/3402137683 shutdown_connections 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9c00640 1 -- 172.21.15.163:0/3402137683 >> 172.21.15.163:0/3402137683 conn(0x7fc6e40b0e30 msgr2=0x7fc6e40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9c00640 1 -- 172.21.15.163:0/3402137683 shutdown_connections 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.139+0000 7fc6e9c00640 1 -- 172.21.15.163:0/3402137683 wait complete. 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.141+0000 7fc6e9c00640 1 Processor -- start 2024-07-26T06:19:06.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.141+0000 7fc6e9c00640 1 -- start start 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e9c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6c8004800 0x7fc6e414c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e9c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 0x7fc6e41527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e9c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6e415c420 0x7fc6e415c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e9c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e4071fe0 con 0x7fc6e415ddc0 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e9c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e4071780 con 0x7fc6e415c420 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e9c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e40b3c50 con 0x7fc6c8004800 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 0x7fc6e41527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 0x7fc6e41527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45726/0 (socket says 172.21.15.163:45726) 2024-07-26T06:19:06.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e3400640 1 -- 172.21.15.163:0/1296797042 learned_addr learned my addr 172.21.15.163:0/1296797042 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:06.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.142+0000 7fc6e3e00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6c8004800 0x7fc6e414c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e9200640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6e415c420 0x7fc6e415c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3400640 1 -- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6c8004800 msgr2=0x7fc6e414c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3400640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6c8004800 0x7fc6e414c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3400640 1 -- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6e415c420 msgr2=0x7fc6e415c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3400640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6e415c420 0x7fc6e415c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3400640 1 -- 172.21.15.163:0/1296797042 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e415e240 con 0x7fc6e415ddc0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3e00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6c8004800 0x7fc6e414c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e3400640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 0x7fc6e41527c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009a70 tx=0x7fc6d0033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0043070 con 0x7fc6e415ddc0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6d0004030 con 0x7fc6e415ddc0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0038040 con 0x7fc6e415ddc0 2024-07-26T06:19:06.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.143+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e415e4c0 con 0x7fc6e415ddc0 2024-07-26T06:19:06.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.144+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e415b940 con 0x7fc6e415ddc0 2024-07-26T06:19:06.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.145+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e40ced60 con 0x7fc6e415ddc0 2024-07-26T06:19:06.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.146+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fc6d0033840 con 0x7fc6e415ddc0 2024-07-26T06:19:06.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.147+0000 7fc6dbe00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc6c4076420 0x7fc6c40788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:06.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.147+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5767+0+0 (secure 0 0 0) 0x7fc6d00b79f0 con 0x7fc6e415ddc0 2024-07-26T06:19:06.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.147+0000 7fc6e3e00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc6c4076420 0x7fc6c40788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:06.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.148+0000 7fc6e3e00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc6c4076420 0x7fc6c40788e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40018e0 tx=0x7fc6d4009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.151+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc6e415e240 con 0x7fc6e415ddc0 2024-07-26T06:19:06.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.405+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc6e40b08c0 con 0x7fc6e415ddc0 2024-07-26T06:19:06.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.406+0000 7fc6dbe00640 1 -- 172.21.15.163:0/1296797042 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) ==== 53+0+73 (secure 0 0 0) 0x7fc6d00891f0 con 0x7fc6e415ddc0 2024-07-26T06:19:06.406 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:19:06.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc6c4076420 msgr2=0x7fc6c40788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:06.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc6c4076420 0x7fc6c40788e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40018e0 tx=0x7fc6d4009040 comp rx=0 tx=0).stop 2024-07-26T06:19:06.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 msgr2=0x7fc6e41527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:06.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 0x7fc6e41527c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009a70 tx=0x7fc6d0033650 comp rx=0 tx=0).stop 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 shutdown_connections 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc6c4076420 0x7fc6c40788e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6e415c420 0x7fc6e415c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc6e415ddc0 0x7fc6e41527c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 --2- 172.21.15.163:0/1296797042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6c8004800 0x7fc6e414c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 >> 172.21.15.163:0/1296797042 conn(0x7fc6e40b0e30 msgr2=0x7fc6e40ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.408+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 shutdown_connections 2024-07-26T06:19:06.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06.409+0000 7fc6e9c00640 1 -- 172.21.15.163:0/1296797042 wait complete. 2024-07-26T06:19:06.428 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:19:06.428 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:19:06.428 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-07-26T06:19:06.428 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:19:06.429 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:19:06.429 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:19:06.429 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:19:06.429 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:19:06.429 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:19:06.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:06.429 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-07-26T06:19:06.478 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:19:06.479 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42258sec preferred_lft 42258sec 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:19:06.480 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::1c3a:f8ff:fec1:3503/64 scope link 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 56:fc:4e:c7:3e:81 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::54fc:4eff:fec7:3e81/64 scope link 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether c2:4f:6d:14:5d:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::c04f:6dff:fe14:5d96/64 scope link 2024-07-26T06:19:06.481 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:06.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:06.482 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:19:06.562 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-07-26T06:19:06.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:06.566 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-07-26T06:19:06.618 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-07-26T06:19:06.619 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:19:06.619 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:19:06.619 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:19:06.619 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:19:06.619 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:19:06.619 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-07-26T06:19:06.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:19:06.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:19:06.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:19:06.619 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-07-26T06:19:06.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:19:06.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:19:06.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:06.821 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:19:06.872 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:19:06.872 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:19:06.873 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:19:06.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:06.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-07-26T06:19:06.954 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:19:06.983 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:19:07.040 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:19:07.040 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:19:07.040 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:19:07.093 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:19:07.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:07.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:19:07.221 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:19:07.222 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:19:07.222 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:19:07.246 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:19:07.246 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:19:07.246 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:19:07.246 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:19:07.246 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:19:07.272 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:19:07.273 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:19:07.274 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:19:07.274 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:19:07.274 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:19:07.274 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:19:07.274 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:19:07.277 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:19:07.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:07.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:19:07.347 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:19:07.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:19:07.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:19:07.478 INFO:teuthology.orchestra.run.smithi163.stdout:client5357 2024-07-26T06:19:07.479 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:07.479 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:07.528 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-07-26T06:19:07.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 -- 172.21.15.163:0/3682762120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf80cb7e0 msgr2=0x7f3cf815ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:07.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/3682762120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf80cb7e0 0x7f3cf815ed20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8005fd0 tx=0x7f3ce802a540 comp rx=0 tx=0).stop 2024-07-26T06:19:07.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 -- 172.21.15.163:0/3682762120 shutdown_connections 2024-07-26T06:19:07.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/3682762120 >> v2:172.21.15.164:3300/0 conn(0x7f3cf8071cf0 0x7f3cf815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:07.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/3682762120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf80cb7e0 0x7f3cf815ed20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:07.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 -- 172.21.15.163:0/3682762120 >> 172.21.15.163:0/3682762120 conn(0x7f3cf806d420 msgr2=0x7f3cf806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:07.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 -- 172.21.15.163:0/3682762120 shutdown_connections 2024-07-26T06:19:07.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.780+0000 7f3cfdc00640 1 -- 172.21.15.163:0/3682762120 wait complete. 2024-07-26T06:19:07.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.782+0000 7f3cfdc00640 1 Processor -- start 2024-07-26T06:19:07.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.782+0000 7f3cfdc00640 1 -- start start 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cfdc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 0x7f3cf8153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cfdc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3cf80cb7e0 0x7f3cf8154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cfdc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf815c380 0x7f3cf80ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cfdc00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf80736b0 con 0x7f3cf8071cf0 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cfdc00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf8073530 con 0x7f3cf80cb7e0 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cfdc00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf8073830 con 0x7f3cf815c380 2024-07-26T06:19:07.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cf6a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 0x7f3cf8153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:07.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cf6a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 0x7f3cf8153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45736/0 (socket says 172.21.15.163:45736) 2024-07-26T06:19:07.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.783+0000 7f3cf6a00640 1 -- 172.21.15.163:0/2540472630 learned_addr learned my addr 172.21.15.163:0/2540472630 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:07.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf7400640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf815c380 0x7f3cf80ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:07.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cee000640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3cf80cb7e0 0x7f3cf8154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf6a00640 1 -- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf815c380 msgr2=0x7f3cf80ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf6a00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf815c380 0x7f3cf80ab400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf6a00640 1 -- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3cf80cb7e0 msgr2=0x7f3cf8154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf6a00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3cf80cb7e0 0x7f3cf8154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf6a00640 1 -- 172.21.15.163:0/2540472630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf8141f30 con 0x7f3cf8071cf0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cf6a00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 0x7f3cf8153f20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8006010 tx=0x7f3ce80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.784+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce803f070 con 0x7f3cf8071cf0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.785+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ce80047a0 con 0x7f3cf8071cf0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.785+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8035030 con 0x7f3cf8071cf0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.785+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf8142210 con 0x7f3cf8071cf0 2024-07-26T06:19:07.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.785+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf81427d0 con 0x7f3cf8071cf0 2024-07-26T06:19:07.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.786+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf8073d50 con 0x7f3cf8071cf0 2024-07-26T06:19:07.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.787+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f3ce8042070 con 0x7f3cf8071cf0 2024-07-26T06:19:07.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.788+0000 7f3cef400640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3cd40763f0 0x7f3cd40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:07.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.788+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f3ce80b8870 con 0x7f3cf8071cf0 2024-07-26T06:19:07.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.788+0000 7f3cee000640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3cd40763f0 0x7f3cd40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:07.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.789+0000 7f3cee000640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3cd40763f0 0x7f3cd40788b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8069ff0 tx=0x7f3ce0005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:07.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:07.793+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3ce8081ed0 con 0x7f3cf8071cf0 2024-07-26T06:19:08.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.040+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3cf8069050 con 0x7f3cf8071cf0 2024-07-26T06:19:08.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.042+0000 7f3cef400640 1 -- 172.21.15.163:0/2540472630 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) ==== 53+0+73 (secure 0 0 0) 0x7f3ce80817d0 con 0x7f3cf8071cf0 2024-07-26T06:19:08.043 INFO:teuthology.orchestra.run.smithi163.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:19:08.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.044+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3cd40763f0 msgr2=0x7f3cd40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:08.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.044+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3cd40763f0 0x7f3cd40788b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8069ff0 tx=0x7f3ce0005f90 comp rx=0 tx=0).stop 2024-07-26T06:19:08.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 msgr2=0x7f3cf8153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:08.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 0x7f3cf8153f20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8006010 tx=0x7f3ce80038a0 comp rx=0 tx=0).stop 2024-07-26T06:19:08.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 shutdown_connections 2024-07-26T06:19:08.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3cd40763f0 0x7f3cd40788b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:08.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3cf815c380 0x7f3cf80ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:08.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3cf80cb7e0 0x7f3cf8154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:08.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 --2- 172.21.15.163:0/2540472630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3cf8071cf0 0x7f3cf8153f20 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:08.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 >> 172.21.15.163:0/2540472630 conn(0x7f3cf806d420 msgr2=0x7f3cf80af780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:08.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 shutdown_connections 2024-07-26T06:19:08.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:08.045+0000 7f3cfdc00640 1 -- 172.21.15.163:0/2540472630 wait complete. 2024-07-26T06:19:08.066 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T06:19:08.066 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:19:08.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-07-26T06:19:08.066 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-07-26T06:19:08.066 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-07-26T06:19:08.067 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T06:19:08.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-07-26T06:19:08.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:19:08.067 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:19:08.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.067 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-07-26T06:19:08.128 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42235sec preferred_lft 42235sec 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:19:08.129 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::4486:3fff:fe0f:8676/64 scope link 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether fe:5b:92:33:7f:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::fc5b:92ff:fe33:7f2b/64 scope link 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5a:b1:99:e0:48:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::58b1:99ff:fee0:4877/64 scope link 2024-07-26T06:19:08.130 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:19:08.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.132 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:19:08.210 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-07-26T06:19:08.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.215 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-07-26T06:19:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-07-26T06:19:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:19:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T06:19:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T06:19:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:19:08.266 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:19:08.266 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-07-26T06:19:08.266 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T06:19:08.266 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T06:19:08.267 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T06:19:08.267 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-07-26T06:19:08.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:19:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:19:08.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:19:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:19:08.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.474 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:19:08.525 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:19:08.525 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:19:08.525 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:19:08.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.525 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-07-26T06:19:08.604 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:19:08.604 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-07-26T06:19:08.654 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-07-26T06:19:08.654 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:19:08.654 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:19:08.704 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-07-26T06:19:08.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.704 DEBUG:teuthology.orchestra.run.smithi164:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:19:08.827 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:19:08.827 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:19:08.828 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:19:08.849 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:19:08.849 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:19:08.849 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:19:08.849 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:19:08.849 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:invalid new device string format 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:3300,172.21.15.164:3300,172.21.15.164:3301" 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: switching to using v1 address with old syntax 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: resolved to: "172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790" 2024-07-26T06:19:08.876 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.163:6789,172.21.15.164:6789,172.21.15.164:6790:/ 2024-07-26T06:19:08.877 INFO:teuthology.orchestra.run.smithi164.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1639b17-d73f-4cae-9471-025ba01f9572" will pass to kernel 2024-07-26T06:19:08.878 INFO:teuthology.orchestra.run.smithi164.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-07-26T06:19:08.878 INFO:teuthology.orchestra.run.smithi164.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:19:08.878 INFO:teuthology.orchestra.run.smithi164.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:19:08.878 INFO:teuthology.orchestra.run.smithi164.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:19:08.878 INFO:teuthology.orchestra.run.smithi164.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:19:08.883 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:19:08.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:08.883 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-07-26T06:19:08.958 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:19:09.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:19:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:19:09.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:19:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:19:09.075 INFO:teuthology.orchestra.run.smithi164.stdout:client5366 2024-07-26T06:19:09.075 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:19:09.075 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:09.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-07-26T06:19:09.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.348+0000 7fa86f400640 1 -- 172.21.15.163:0/1254574489 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868031c30 con 0x7fa8700b4420 2024-07-26T06:19:09.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.348+0000 7fa875a00640 1 -- 172.21.15.163:0/1254574489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700b4420 msgr2=0x7fa8700b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.348+0000 7fa875a00640 1 --2- 172.21.15.163:0/1254574489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700b4420 0x7fa8700b2130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa868005fd0 tx=0x7fa86802a8b0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.348+0000 7fa875a00640 1 -- 172.21.15.163:0/1254574489 shutdown_connections 2024-07-26T06:19:09.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.348+0000 7fa875a00640 1 --2- 172.21.15.163:0/1254574489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700b4420 0x7fa8700b2130 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.348+0000 7fa875a00640 1 --2- 172.21.15.163:0/1254574489 >> v2:172.21.15.164:3300/0 conn(0x7fa8700cbad0 0x7fa870159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.349+0000 7fa875a00640 1 -- 172.21.15.163:0/1254574489 >> 172.21.15.163:0/1254574489 conn(0x7fa8700b0e30 msgr2=0x7fa8700b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:09.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.349+0000 7fa875a00640 1 -- 172.21.15.163:0/1254574489 shutdown_connections 2024-07-26T06:19:09.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.349+0000 7fa875a00640 1 -- 172.21.15.163:0/1254574489 wait complete. 2024-07-26T06:19:09.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.351+0000 7fa875a00640 1 Processor -- start 2024-07-26T06:19:09.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.351+0000 7fa875a00640 1 -- start start 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa875a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700b4420 0x7fa87014b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa875a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 0x7fa8701525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa875a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa870073af0 0x7fa8700b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa875a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa87006f2f0 con 0x7fa870159370 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa875a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8700cd290 con 0x7fa8700b4420 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa875a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa87006f470 con 0x7fa870073af0 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa867e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 0x7fa8701525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa867e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 0x7fa8701525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45740/0 (socket says 172.21.15.163:45740) 2024-07-26T06:19:09.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.352+0000 7fa867e00640 1 -- 172.21.15.163:0/859078627 learned_addr learned my addr 172.21.15.163:0/859078627 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa86fe00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700b4420 0x7fa87014b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa875000640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa870073af0 0x7fa8700b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa867e00640 1 -- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa870073af0 msgr2=0x7fa8700b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa867e00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa870073af0 0x7fa8700b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa867e00640 1 -- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700b4420 msgr2=0x7fa87014b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa867e00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700b4420 0x7fa87014b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa867e00640 1 -- 172.21.15.163:0/859078627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87014be40 con 0x7fa870159370 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa867e00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 0x7fa8701525c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa85800ea20 tx=0x7fa85800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.353+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85800cd80 con 0x7fa870159370 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.354+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa858010430 con 0x7fa870159370 2024-07-26T06:19:09.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.354+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858010840 con 0x7fa870159370 2024-07-26T06:19:09.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.354+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa870155570 con 0x7fa870159370 2024-07-26T06:19:09.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.354+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8700742b0 con 0x7fa870159370 2024-07-26T06:19:09.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.355+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87014c450 con 0x7fa870159370 2024-07-26T06:19:09.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.356+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fa8580109e0 con 0x7fa870159370 2024-07-26T06:19:09.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.357+0000 7fa86cc00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa84c0763f0 0x7fa84c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.357+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7fa858098530 con 0x7fa870159370 2024-07-26T06:19:09.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.357+0000 7fa86fe00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa84c0763f0 0x7fa84c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.358+0000 7fa86fe00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa84c0763f0 0x7fa84c0788b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa870155790 tx=0x7fa860009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:09.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.361+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa858061c30 con 0x7fa870159370 2024-07-26T06:19:09.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.602+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fa8700b2130 con 0x7fa870159370 2024-07-26T06:19:09.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.657+0000 7fa86cc00640 1 -- 172.21.15.163:0/859078627 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v71) ==== 136+0+0 (secure 0 0 0) 0x7fa858061530 con 0x7fa870159370 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.659+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa84c0763f0 msgr2=0x7fa84c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa84c0763f0 0x7fa84c0788b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa870155790 tx=0x7fa860009040 comp rx=0 tx=0).stop 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 msgr2=0x7fa8701525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 0x7fa8701525c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa85800ea20 tx=0x7fa85800eee0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 shutdown_connections 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fa84c0763f0 0x7fa84c0788b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa870073af0 0x7fa8700b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa870159370 0x7fa8701525c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 --2- 172.21.15.163:0/859078627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700b4420 0x7fa87014b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 >> 172.21.15.163:0/859078627 conn(0x7fa8700b0e30 msgr2=0x7fa8700ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:09.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 shutdown_connections 2024-07-26T06:19:09.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.660+0000 7fa875a00640 1 -- 172.21.15.163:0/859078627 wait complete. 2024-07-26T06:19:09.680 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-07-26T06:19:09.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 -- 172.21.15.163:0/1591881303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf80b1e80 msgr2=0x7f2bf80b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/1591881303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf80b1e80 0x7f2bf80b6bd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0005fd0 tx=0x7f2bf002a410 comp rx=0 tx=0).stop 2024-07-26T06:19:09.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 -- 172.21.15.163:0/1591881303 shutdown_connections 2024-07-26T06:19:09.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/1591881303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf80b1e80 0x7f2bf80b6bd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/1591881303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf815d5a0 0x7f2bf80ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 -- 172.21.15.163:0/1591881303 >> 172.21.15.163:0/1591881303 conn(0x7f2bf80cfda0 msgr2=0x7f2bf80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:09.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 -- 172.21.15.163:0/1591881303 shutdown_connections 2024-07-26T06:19:09.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.877+0000 7f2bfcc00640 1 -- 172.21.15.163:0/1591881303 wait complete. 2024-07-26T06:19:09.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.879+0000 7f2bfcc00640 1 Processor -- start 2024-07-26T06:19:09.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.879+0000 7f2bfcc00640 1 -- start start 2024-07-26T06:19:09.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bfcc00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf80b1e80 0x7f2bf815daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bfcc00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf80ce830 0x7f2bf814eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bfcc00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 0x7f2bf80ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bfcc00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf80b5130 con 0x7f2bf815d5a0 2024-07-26T06:19:09.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bfcc00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf80b4fb0 con 0x7f2bf80ce830 2024-07-26T06:19:09.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bfcc00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf80b52b0 con 0x7f2bf80b1e80 2024-07-26T06:19:09.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 0x7f2bf80ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 0x7f2bf80ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45752/0 (socket says 172.21.15.163:45752) 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 -- 172.21.15.163:0/3197801961 learned_addr learned my addr 172.21.15.163:0/3197801961 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 -- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf80b1e80 msgr2=0x7f2bf815daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf6a00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf80ce830 0x7f2bf814eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf80b1e80 0x7f2bf815daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 -- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf80ce830 msgr2=0x7f2bf814eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf80ce830 0x7f2bf814eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.880+0000 7f2bf7e00640 1 -- 172.21.15.163:0/3197801961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf81f7ea0 con 0x7f2bf815d5a0 2024-07-26T06:19:09.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.881+0000 7f2bf7e00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 0x7f2bf80ca3f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006010 tx=0x7f2bf0003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:09.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.881+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf003f070 con 0x7f2bf815d5a0 2024-07-26T06:19:09.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.881+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bf00047a0 con 0x7f2bf815d5a0 2024-07-26T06:19:09.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.881+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0035030 con 0x7f2bf815d5a0 2024-07-26T06:19:09.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.881+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf80cbe00 con 0x7f2bf815d5a0 2024-07-26T06:19:09.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.881+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf81f8140 con 0x7f2bf815d5a0 2024-07-26T06:19:09.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.882+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf80ce2f0 con 0x7f2bf815d5a0 2024-07-26T06:19:09.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.883+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f2bf0042070 con 0x7f2bf815d5a0 2024-07-26T06:19:09.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.884+0000 7f2bef400640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2bd40763f0 0x7f2bd40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:09.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.884+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f2bf00b88e0 con 0x7f2bf815d5a0 2024-07-26T06:19:09.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.885+0000 7f2bf7400640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2bd40763f0 0x7f2bd40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:09.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.885+0000 7f2bf7400640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2bd40763f0 0x7f2bd40788b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2be80058c0 tx=0x7f2be8005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:09.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:09.889+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2bf0081fd0 con 0x7f2bf815d5a0 2024-07-26T06:19:10.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.101+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f2bf8160280 con 0x7f2bd40763f0 2024-07-26T06:19:10.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.104+0000 7f2bef400640 1 -- 172.21.15.163:0/3197801961 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f2bf8160280 con 0x7f2bd40763f0 2024-07-26T06:19:10.105 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:19:10.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:19:10.105 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "cephfs" 2024-07-26T06:19:10.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:19:10.105 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:19:10.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2bd40763f0 msgr2=0x7f2bd40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2bd40763f0 0x7f2bd40788b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2be80058c0 tx=0x7f2be8005850 comp rx=0 tx=0).stop 2024-07-26T06:19:10.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 msgr2=0x7f2bf80ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 0x7f2bf80ca3f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006010 tx=0x7f2bf0003890 comp rx=0 tx=0).stop 2024-07-26T06:19:10.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 shutdown_connections 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2bd40763f0 0x7f2bd40788b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2bf815d5a0 0x7f2bf80ca3f0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf80ce830 0x7f2bf814eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 --2- 172.21.15.163:0/3197801961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf80b1e80 0x7f2bf815daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.107+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 >> 172.21.15.163:0/3197801961 conn(0x7f2bf80cfda0 msgr2=0x7f2bf80b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.108+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 shutdown_connections 2024-07-26T06:19:10.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.108+0000 7f2bfcc00640 1 -- 172.21.15.163:0/3197801961 wait complete. 2024-07-26T06:19:10.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.335+0000 7f8a38c00640 1 -- 172.21.15.163:0/2593589027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 msgr2=0x7f8a34071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.335+0000 7f8a38c00640 1 --2- 172.21.15.163:0/2593589027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a34071530 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009830 tx=0x7f8a1c02a510 comp rx=0 tx=0).stop 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.336+0000 7f8a38c00640 1 -- 172.21.15.163:0/2593589027 shutdown_connections 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.336+0000 7f8a38c00640 1 --2- 172.21.15.163:0/2593589027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a34071530 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.336+0000 7f8a38c00640 1 --2- 172.21.15.163:0/2593589027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a341556f0 0x7f8a340cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.336+0000 7f8a38c00640 1 -- 172.21.15.163:0/2593589027 >> 172.21.15.163:0/2593589027 conn(0x7f8a340b0e30 msgr2=0x7f8a340b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.336+0000 7f8a38c00640 1 -- 172.21.15.163:0/2593589027 shutdown_connections 2024-07-26T06:19:10.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.336+0000 7f8a38c00640 1 -- 172.21.15.163:0/2593589027 wait complete. 2024-07-26T06:19:10.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.337+0000 7f8a38c00640 1 Processor -- start 2024-07-26T06:19:10.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.337+0000 7f8a38c00640 1 -- start start 2024-07-26T06:19:10.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.338+0000 7f8a38c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a34071a70 0x7f8a3413da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:10.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.338+0000 7f8a38c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a340aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.338+0000 7f8a38c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a341556f0 0x7f8a340af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.338+0000 7f8a38c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3415f310 con 0x7f8a340ce2d0 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.338+0000 7f8a38c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34071ee0 con 0x7f8a341556f0 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.338+0000 7f8a38c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3415f490 con 0x7f8a34071a70 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a340aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a340aeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45766/0 (socket says 172.21.15.163:45766) 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 -- 172.21.15.163:0/115444218 learned_addr learned my addr 172.21.15.163:0/115444218 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 -- 172.21.15.163:0/115444218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a34071a70 msgr2=0x7f8a3413da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a34071a70 0x7f8a3413da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 -- 172.21.15.163:0/115444218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a341556f0 msgr2=0x7f8a340af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a341556f0 0x7f8a340af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 -- 172.21.15.163:0/115444218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3414e8c0 con 0x7f8a340ce2d0 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a32a00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a340aeeb0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c02ab00 tx=0x7f8a1c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.339+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c040070 con 0x7f8a340ce2d0 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.340+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a1c005310 con 0x7f8a340ce2d0 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.340+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c005c90 con 0x7f8a340ce2d0 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.340+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3414eba0 con 0x7f8a340ce2d0 2024-07-26T06:19:10.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.340+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3414f160 con 0x7f8a340ce2d0 2024-07-26T06:19:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.341+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a34071530 con 0x7f8a340ce2d0 2024-07-26T06:19:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.342+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f8a1c039040 con 0x7f8a340ce2d0 2024-07-26T06:19:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.343+0000 7f8a2b400640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8a100763f0 0x7f8a100788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.343+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f8a1c0b8d30 con 0x7f8a340ce2d0 2024-07-26T06:19:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.343+0000 7f8a33400640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8a100763f0 0x7f8a100788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.344+0000 7f8a33400640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8a100763f0 0x7f8a100788b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a240058c0 tx=0x7f8a24005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:10.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.347+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8a1c082390 con 0x7f8a340ce2d0 2024-07-26T06:19:10.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.554+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8a340b02b0 con 0x7f8a340ce2d0 2024-07-26T06:19:10.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.555+0000 7f8a2b400640 1 -- 172.21.15.163:0/115444218 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f8a1c081c90 con 0x7f8a340ce2d0 2024-07-26T06:19:10.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.557+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8a100763f0 msgr2=0x7f8a100788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.557+0000 7f8a38c00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8a100763f0 0x7f8a100788b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a240058c0 tx=0x7f8a24005850 comp rx=0 tx=0).stop 2024-07-26T06:19:10.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.557+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 msgr2=0x7f8a340aeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:10.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.557+0000 7f8a38c00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a340aeeb0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c02ab00 tx=0x7f8a1c032c10 comp rx=0 tx=0).stop 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 shutdown_connections 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f8a100763f0 0x7f8a100788b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a341556f0 0x7f8a340af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a340ce2d0 0x7f8a340aeeb0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 --2- 172.21.15.163:0/115444218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a34071a70 0x7f8a3413da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 >> 172.21.15.163:0/115444218 conn(0x7f8a340b0e30 msgr2=0x7f8a340b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 shutdown_connections 2024-07-26T06:19:10.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:10.558+0000 7f8a38c00640 1 -- 172.21.15.163:0/115444218 wait complete. 2024-07-26T06:19:10.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:10.578 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b) 2024-07-26T06:19:10.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:10.663 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/.snap/fake) 2024-07-26T06:19:10.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:10.782 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-07-26T06:19:10.835 INFO:teuthology.orchestra.run.smithi163.stdout:cephfs_data 2024-07-26T06:19:10.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:10.835 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b) 2024-07-26T06:19:10.920 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:10.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol99070/.meta 2024-07-26T06:19:11.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol99070 2024-07-26T06:19:11.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 -- 172.21.15.163:0/3731297163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda180052e0 msgr2=0x7fda1800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 --2- 172.21.15.163:0/3731297163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda180052e0 0x7fda1800b9c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fda34072190 tx=0x7fda18005830 comp rx=0 tx=0).stop 2024-07-26T06:19:11.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 -- 172.21.15.163:0/3731297163 shutdown_connections 2024-07-26T06:19:11.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 --2- 172.21.15.163:0/3731297163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda180052e0 0x7fda1800b9c0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 -- 172.21.15.163:0/3731297163 >> 172.21.15.163:0/3731297163 conn(0x7fda340b1ee0 msgr2=0x7fda340b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:11.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 -- 172.21.15.163:0/3731297163 shutdown_connections 2024-07-26T06:19:11.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.267+0000 7fda39400640 1 -- 172.21.15.163:0/3731297163 wait complete. 2024-07-26T06:19:11.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.269+0000 7fda39400640 1 Processor -- start 2024-07-26T06:19:11.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.269+0000 7fda39400640 1 -- start start 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda39400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda180052e0 0x7fda341f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda39400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 0x7fda34145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda39400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda340a94d0 0x7fda340ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda39400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fda3415eae0 con 0x7fda341f3180 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda39400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fda340cd7c0 con 0x7fda340a94d0 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda39400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fda3415ec60 con 0x7fda180052e0 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda33400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 0x7fda34145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda33400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 0x7fda34145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45798/0 (socket says 172.21.15.163:45798) 2024-07-26T06:19:11.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda33400640 1 -- 172.21.15.163:0/1642376516 learned_addr learned my addr 172.21.15.163:0/1642376516 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda38a00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda340a94d0 0x7fda340ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.270+0000 7fda33e00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda180052e0 0x7fda341f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33400640 1 -- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda180052e0 msgr2=0x7fda341f1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda180052e0 0x7fda341f1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33400640 1 -- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda340a94d0 msgr2=0x7fda340ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda340a94d0 0x7fda340ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33400640 1 -- 172.21.15.163:0/1642376516 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda341f3600 con 0x7fda341f3180 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda38a00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda340a94d0 0x7fda340ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33e00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda180052e0 0x7fda341f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:11.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda33400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 0x7fda34145a00 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00c430 tx=0x7fda1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c017070 con 0x7fda341f3180 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda1c0053c0 con 0x7fda341f3180 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda34145f40 con 0x7fda341f3180 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c003850 con 0x7fda341f3180 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.271+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda341f3b60 con 0x7fda341f3180 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.272+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3415cdc0 con 0x7fda341f3180 2024-07-26T06:19:11.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.274+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7fda1c01a070 con 0x7fda341f3180 2024-07-26T06:19:11.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.275+0000 7fda2be00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fda10076420 0x7fda100788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.275+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7fda1c096ef0 con 0x7fda341f3180 2024-07-26T06:19:11.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.275+0000 7fda33e00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fda10076420 0x7fda100788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:11.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.276+0000 7fda33e00640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fda10076420 0x7fda100788e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda2c001860 tx=0x7fda2c001c20 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:11.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.279+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda341f3600 con 0x7fda341f3180 2024-07-26T06:19:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.515+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol99070", "target": ["mon-mgr", ""]}) -- 0x7fda340cc560 con 0x7fda10076420 2024-07-26T06:19:11.521 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:11.521+0000 7fc4c0c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:11.521 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:11.522+0000 7fc4c0c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:11.522 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:11.522+0000 7fc4c0c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:11.522 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:11.522+0000 7fc4c0c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:11.522 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:11.522+0000 7fc4c0c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:11.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.578+0000 7fda2be00640 1 -- 172.21.15.163:0/1642376516 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7fda340cc560 con 0x7fda10076420 2024-07-26T06:19:11.579 INFO:teuthology.orchestra.run.smithi163.stdout:/volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b 2024-07-26T06:19:11.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fda10076420 msgr2=0x7fda100788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fda10076420 0x7fda100788e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda2c001860 tx=0x7fda2c001c20 comp rx=0 tx=0).stop 2024-07-26T06:19:11.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 msgr2=0x7fda34145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 0x7fda34145a00 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00c430 tx=0x7fda1c00c8f0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 shutdown_connections 2024-07-26T06:19:11.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fda10076420 0x7fda100788e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda340a94d0 0x7fda340ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fda341f3180 0x7fda34145a00 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 --2- 172.21.15.163:0/1642376516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda180052e0 0x7fda341f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 >> 172.21.15.163:0/1642376516 conn(0x7fda340b1ee0 msgr2=0x7fda340cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 shutdown_connections 2024-07-26T06:19:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.581+0000 7fda39400640 1 -- 172.21.15.163:0/1642376516 wait complete. 2024-07-26T06:19:11.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-07-26T06:19:11.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 -- 172.21.15.163:0/2760351970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 msgr2=0x7f3a9c0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 --2- 172.21.15.163:0/2760351970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c0b3710 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3a94005fd0 tx=0x7f3a9402a8b0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 -- 172.21.15.163:0/2760351970 shutdown_connections 2024-07-26T06:19:11.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 --2- 172.21.15.163:0/2760351970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c0b3710 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 --2- 172.21.15.163:0/2760351970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a9c0ce4a0 0x7f3a9c071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 -- 172.21.15.163:0/2760351970 >> 172.21.15.163:0/2760351970 conn(0x7f3a9c0b0e30 msgr2=0x7f3a9c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:11.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 -- 172.21.15.163:0/2760351970 shutdown_connections 2024-07-26T06:19:11.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.793+0000 7f3aa2400640 1 -- 172.21.15.163:0/2760351970 wait complete. 2024-07-26T06:19:11.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.795+0000 7f3aa2400640 1 Processor -- start 2024-07-26T06:19:11.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.795+0000 7f3aa2400640 1 -- start start 2024-07-26T06:19:11.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa2400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c1e8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa2400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a9c0ce4a0 0x7f3a9c0a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa2400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a9c1556f0 0x7f3a9c150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa2400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c0b4020 con 0x7f3a9c071a70 2024-07-26T06:19:11.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa2400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c0b3ea0 con 0x7f3a9c0ce4a0 2024-07-26T06:19:11.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa2400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9c0b41a0 con 0x7f3a9c1556f0 2024-07-26T06:19:11.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa1000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c1e8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:11.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa1000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c1e8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45836/0 (socket says 172.21.15.163:45836) 2024-07-26T06:19:11.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.796+0000 7f3aa1000640 1 -- 172.21.15.163:0/327134887 learned_addr learned my addr 172.21.15.163:0/327134887 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:11.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa1000640 1 -- 172.21.15.163:0/327134887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a9c1556f0 msgr2=0x7f3a9c150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa1000640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a9c1556f0 0x7f3a9c150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa1000640 1 -- 172.21.15.163:0/327134887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a9c0ce4a0 msgr2=0x7f3a9c0a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:11.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa1000640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a9c0ce4a0 0x7f3a9c0a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:11.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa1000640 1 -- 172.21.15.163:0/327134887 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c150b70 con 0x7f3a9c071a70 2024-07-26T06:19:11.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa1000640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c1e8800 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000c430 tx=0x7f3a9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:11.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90011790 con 0x7f3a9c071a70 2024-07-26T06:19:11.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a90017070 con 0x7f3a9c071a70 2024-07-26T06:19:11.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90011ce0 con 0x7f3a9c071a70 2024-07-26T06:19:11.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9c1e8f60 con 0x7f3a9c071a70 2024-07-26T06:19:11.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.797+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9c1e9520 con 0x7f3a9c071a70 2024-07-26T06:19:11.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.798+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a9c0b4730 con 0x7f3a9c071a70 2024-07-26T06:19:11.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.800+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f3a9001c460 con 0x7f3a9c071a70 2024-07-26T06:19:11.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.801+0000 7f3a98c00640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3a7c0763f0 0x7f3a7c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:11.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.801+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f3a90097720 con 0x7f3a9c071a70 2024-07-26T06:19:11.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.801+0000 7f3a9be00640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3a7c0763f0 0x7f3a7c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:11.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.802+0000 7f3a9be00640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3a7c0763f0 0x7f3a7c0788b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3a88005fd0 tx=0x7f3a88005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:11.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:11.805+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a90060f30 con 0x7f3a9c071a70 2024-07-26T06:19:12.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.042+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7f3a9c0b34f0 con 0x7f3a7c0763f0 2024-07-26T06:19:12.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.050+0000 7f3a98c00640 1 -- 172.21.15.163:0/327134887 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f3a9c0b34f0 con 0x7f3a7c0763f0 2024-07-26T06:19:12.050 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:19:12.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:19:12.050 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "subvol99070" 2024-07-26T06:19:12.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:19:12.051 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:19:12.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3a7c0763f0 msgr2=0x7f3a7c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3a7c0763f0 0x7f3a7c0788b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3a88005fd0 tx=0x7f3a88005850 comp rx=0 tx=0).stop 2024-07-26T06:19:12.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 msgr2=0x7f3a9c1e8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c1e8800 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000c430 tx=0x7f3a9000c8f0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 shutdown_connections 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f3a7c0763f0 0x7f3a7c0788b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a9c1556f0 0x7f3a9c150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a9c0ce4a0 0x7f3a9c0a9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 --2- 172.21.15.163:0/327134887 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3a9c071a70 0x7f3a9c1e8800 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 >> 172.21.15.163:0/327134887 conn(0x7f3a9c0b0e30 msgr2=0x7f3a9c151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.053+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 shutdown_connections 2024-07-26T06:19:12.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.054+0000 7f3aa2400640 1 -- 172.21.15.163:0/327134887 wait complete. 2024-07-26T06:19:12.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol99070 2024-07-26T06:19:12.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 -- 172.21.15.163:0/492458009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 msgr2=0x7f28f40cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 --2- 172.21.15.163:0/492458009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f40cdd90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f28e4005fd0 tx=0x7f28e402a530 comp rx=0 tx=0).stop 2024-07-26T06:19:12.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 -- 172.21.15.163:0/492458009 shutdown_connections 2024-07-26T06:19:12.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 --2- 172.21.15.163:0/492458009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28f40ce2d0 0x7f28f4071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 --2- 172.21.15.163:0/492458009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f40cdd90 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 -- 172.21.15.163:0/492458009 >> 172.21.15.163:0/492458009 conn(0x7f28f40b0e30 msgr2=0x7f28f40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:12.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 -- 172.21.15.163:0/492458009 shutdown_connections 2024-07-26T06:19:12.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.276+0000 7f28f3e00640 1 -- 172.21.15.163:0/492458009 wait complete. 2024-07-26T06:19:12.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.278+0000 7f28f3e00640 1 Processor -- start 2024-07-26T06:19:12.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.278+0000 7f28f3e00640 1 -- start start 2024-07-26T06:19:12.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28f4071a70 0x7f28f4145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28f40ce2d0 0x7f28f414fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f41f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28f415f310 con 0x7f28f41556f0 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4071ee0 con 0x7f28f40ce2d0 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f28f415f490 con 0x7f28f4071a70 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f41f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f41f40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45866/0 (socket says 172.21.15.163:45866) 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f3400640 1 -- 172.21.15.163:0/3822398410 learned_addr learned my addr 172.21.15.163:0/3822398410 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28f2a00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28f4071a70 0x7f28f4145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.279+0000 7f28ebe00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28f40ce2d0 0x7f28f414fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3400640 1 -- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28f4071a70 msgr2=0x7f28f4145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3400640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28f4071a70 0x7f28f4145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3400640 1 -- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28f40ce2d0 msgr2=0x7f28f414fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3400640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28f40ce2d0 0x7f28f414fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3400640 1 -- 172.21.15.163:0/3822398410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f414d640 con 0x7f28f41556f0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3400640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f41f40b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00c780 tx=0x7f28ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec004b60 con 0x7f28f41556f0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28ec004df0 con 0x7f28f41556f0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00aa90 con 0x7f28f41556f0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f414d920 con 0x7f28f41556f0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.280+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f414e1c0 con 0x7f28f41556f0 2024-07-26T06:19:12.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.282+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f40ced60 con 0x7f28f41556f0 2024-07-26T06:19:12.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.283+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 98303+0+0 (secure 0 0 0) 0x7f28ec00ac30 con 0x7f28f41556f0 2024-07-26T06:19:12.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.284+0000 7f28eaa00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f28d00763f0 0x7f28d00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.284+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f28ec097bc0 con 0x7f28f41556f0 2024-07-26T06:19:12.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.284+0000 7f28f2a00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f28d00763f0 0x7f28d00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.285+0000 7f28f2a00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f28d00763f0 0x7f28d00788b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f28e4033000 tx=0x7f28e4004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:12.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.288+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f28ec020070 con 0x7f28f41556f0 2024-07-26T06:19:12.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.374+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f28ec05cff0 con 0x7f28f41556f0 2024-07-26T06:19:12.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.517+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol99070", "target": ["mon-mgr", ""]}) -- 0x7f28f414dd20 con 0x7f28d00763f0 2024-07-26T06:19:12.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.575+0000 7f28eaa00640 1 -- 172.21.15.163:0/3822398410 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+687 (secure 0 0 0) 0x7f28e4004340 con 0x7f28d00763f0 2024-07-26T06:19:12.575 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "atime": "2024-07-26 06:19:10", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_pcent": "undefined", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_quota": "infinite", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_used": 0, 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "created_at": "2024-07-26 06:19:10", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "ctime": "2024-07-26 06:19:11", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "data_pool": "cephfs_data", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "features": [ 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot-clone", 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot-autoprotect" 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "flavor": 1, 2024-07-26T06:19:12.576 INFO:teuthology.orchestra.run.smithi163.stdout: "gid": 0, 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mode": 16877, 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_addrs": [ 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6789", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.164:6789", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.164:6790" 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mtime": "2024-07-26 06:19:10", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "path": "/volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_namespace": "", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "complete", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "subvolume", 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout: "uid": 0 2024-07-26T06:19:12.577 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:12.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f28d00763f0 msgr2=0x7f28d00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f28d00763f0 0x7f28d00788b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f28e4033000 tx=0x7f28e4004730 comp rx=0 tx=0).stop 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 msgr2=0x7f28f41f40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f41f40b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00c780 tx=0x7f28ec00cc40 comp rx=0 tx=0).stop 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 shutdown_connections 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f28d00763f0 0x7f28d00788b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28f41556f0 0x7f28f41f40b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28f40ce2d0 0x7f28f414fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.578+0000 7f28f3e00640 1 --2- 172.21.15.163:0/3822398410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28f4071a70 0x7f28f4145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.579+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 >> 172.21.15.163:0/3822398410 conn(0x7f28f40b0e30 msgr2=0x7f28f40b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.579+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 shutdown_connections 2024-07-26T06:19:12.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.579+0000 7f28f3e00640 1 -- 172.21.15.163:0/3822398410 wait complete. 2024-07-26T06:19:12.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize cephfs subvol99070 10485760 2024-07-26T06:19:12.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 -- 172.21.15.163:0/4089781509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c140800 msgr2=0x7f496c071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 --2- 172.21.15.163:0/4089781509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c140800 0x7f496c071200 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4958009830 tx=0x7f495802a510 comp rx=0 tx=0).stop 2024-07-26T06:19:12.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 -- 172.21.15.163:0/4089781509 shutdown_connections 2024-07-26T06:19:12.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 --2- 172.21.15.163:0/4089781509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c140800 0x7f496c071200 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 -- 172.21.15.163:0/4089781509 >> 172.21.15.163:0/4089781509 conn(0x7f496c0b0e30 msgr2=0x7f496c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:12.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 -- 172.21.15.163:0/4089781509 shutdown_connections 2024-07-26T06:19:12.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.804+0000 7f496b200640 1 -- 172.21.15.163:0/4089781509 wait complete. 2024-07-26T06:19:12.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.806+0000 7f496b200640 1 Processor -- start 2024-07-26T06:19:12.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.806+0000 7f496b200640 1 -- start start 2024-07-26T06:19:12.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f496c073530 0x7f496c159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f496c140800 0x7f496c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 0x7f496c0b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f496c166450 con 0x7f496c141150 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1662d0 con 0x7f496c140800 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f496c1665d0 con 0x7f496c073530 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 0x7f496c0b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 0x7f496c0b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45888/0 (socket says 172.21.15.163:45888) 2024-07-26T06:19:12.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.807+0000 7f496a800640 1 -- 172.21.15.163:0/1558104680 learned_addr learned my addr 172.21.15.163:0/1558104680 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f4969400640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f496c140800 0x7f496c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496a800640 1 -- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f496c073530 msgr2=0x7f496c159b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496a800640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f496c073530 0x7f496c159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496a800640 1 -- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f496c140800 msgr2=0x7f496c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496a800640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f496c140800 0x7f496c0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496a800640 1 -- 172.21.15.163:0/1558104680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c15a400 con 0x7f496c141150 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f4969400640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f496c140800 0x7f496c0b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496a800640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 0x7f496c0b4a10 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f496400c780 tx=0x7f496400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964016020 con 0x7f496c141150 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f496400a930 con 0x7f496c141150 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c15a6e0 con 0x7f496c141150 2024-07-26T06:19:12.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496400adc0 con 0x7f496c141150 2024-07-26T06:19:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.808+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c1567d0 con 0x7f496c141150 2024-07-26T06:19:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.809+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c071200 con 0x7f496c141150 2024-07-26T06:19:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.811+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f4964019020 con 0x7f496c141150 2024-07-26T06:19:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.812+0000 7f4962000640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f494c076480 0x7f494c078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.812+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f49640979f0 con 0x7f496c141150 2024-07-26T06:19:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.812+0000 7f4969e00640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f494c076480 0x7f494c078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.813+0000 7f4969e00640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f494c076480 0x7f494c078940 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f495c007fd0 tx=0x7f495c007900 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:12.816+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4964020030 con 0x7f496c141150 2024-07-26T06:19:13.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.059+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "cephfs", "sub_name": "subvol99070", "new_size": "10485760", "target": ["mon-mgr", ""]}) -- 0x7f496c0b4f50 con 0x7f494c076480 2024-07-26T06:19:13.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.128+0000 7f4962000640 1 -- 172.21.15.163:0/1558104680 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+128 (secure 0 0 0) 0x7f496c0b4f50 con 0x7f494c076480 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout:[ 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_used": 0 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_quota": 10485760 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_pcent": "0.00" 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-07-26T06:19:13.129 INFO:teuthology.orchestra.run.smithi163.stdout:] 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f494c076480 msgr2=0x7f494c078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f494c076480 0x7f494c078940 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f495c007fd0 tx=0x7f495c007900 comp rx=0 tx=0).stop 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 msgr2=0x7f496c0b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 0x7f496c0b4a10 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f496400c780 tx=0x7f496400cc40 comp rx=0 tx=0).stop 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 shutdown_connections 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f494c076480 0x7f494c078940 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f496c141150 0x7f496c0b4a10 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f496c140800 0x7f496c0b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 --2- 172.21.15.163:0/1558104680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f496c073530 0x7f496c159b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 >> 172.21.15.163:0/1558104680 conn(0x7f496c0b0e30 msgr2=0x7f496c1400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 shutdown_connections 2024-07-26T06:19:13.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.132+0000 7f496b200640 1 -- 172.21.15.163:0/1558104680 wait complete. 2024-07-26T06:19:13.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol99070 2024-07-26T06:19:13.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 -- 172.21.15.163:0/3342519092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff340003a00 msgr2=0x7ff340003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 --2- 172.21.15.163:0/3342519092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff340003a00 0x7ff340003e00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005fd0 tx=0x7ff340005830 comp rx=0 tx=0).stop 2024-07-26T06:19:13.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 -- 172.21.15.163:0/3342519092 shutdown_connections 2024-07-26T06:19:13.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 --2- 172.21.15.163:0/3342519092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff340003a00 0x7ff340003e00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 -- 172.21.15.163:0/3342519092 >> 172.21.15.163:0/3342519092 conn(0x7ff35c0d36b0 msgr2=0x7ff35c0d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:13.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 -- 172.21.15.163:0/3342519092 shutdown_connections 2024-07-26T06:19:13.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.367+0000 7ff361a00640 1 -- 172.21.15.163:0/3342519092 wait complete. 2024-07-26T06:19:13.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.369+0000 7ff361a00640 1 Processor -- start 2024-07-26T06:19:13.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.369+0000 7ff361a00640 1 -- start start 2024-07-26T06:19:13.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff361a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff340003a00 0x7ff35c15e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff361a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff35c160760 0x7ff35c153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff361a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 0x7ff35c154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff361a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c0b5980 con 0x7ff35c154900 2024-07-26T06:19:13.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff361a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c0b5800 con 0x7ff35c160760 2024-07-26T06:19:13.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff361a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff35c0b5b00 con 0x7ff340003a00 2024-07-26T06:19:13.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff35b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 0x7ff35c154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff35b400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 0x7ff35c154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45896/0 (socket says 172.21.15.163:45896) 2024-07-26T06:19:13.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.370+0000 7ff35b400640 1 -- 172.21.15.163:0/256903634 learned_addr learned my addr 172.21.15.163:0/256903634 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:13.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff353e00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff35c160760 0x7ff35c153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35aa00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff340003a00 0x7ff35c15e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35b400640 1 -- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff340003a00 msgr2=0x7ff35c15e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35b400640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff340003a00 0x7ff35c15e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35b400640 1 -- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff35c160760 msgr2=0x7ff35c153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35b400640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff35c160760 0x7ff35c153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35b400640 1 -- 172.21.15.163:0/256903634 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35c160be0 con 0x7ff35c154900 2024-07-26T06:19:13.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.371+0000 7ff35b400640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 0x7ff35c154d80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7ff35c161000 tx=0x7ff35400d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:13.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.376+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35400f880 con 0x7ff35c154900 2024-07-26T06:19:13.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.376+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff354017070 con 0x7ff35c154900 2024-07-26T06:19:13.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.376+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35c15ec80 con 0x7ff35c154900 2024-07-26T06:19:13.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.377+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35400fdb0 con 0x7ff35c154900 2024-07-26T06:19:13.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.377+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35c15f240 con 0x7ff35c154900 2024-07-26T06:19:13.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.378+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff35c0d1990 con 0x7ff35c154900 2024-07-26T06:19:13.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.379+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7ff35401c460 con 0x7ff35c154900 2024-07-26T06:19:13.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.380+0000 7ff352a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff338076500 0x7ff3380789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.380+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7ff354097bf0 con 0x7ff35c154900 2024-07-26T06:19:13.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.381+0000 7ff35aa00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff338076500 0x7ff3380789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.381+0000 7ff35aa00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff338076500 0x7ff3380789c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff34c036000 tx=0x7ff34c0046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:13.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.385+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff354061160 con 0x7ff35c154900 2024-07-26T06:19:13.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.622+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol99070", "target": ["mon-mgr", ""]}) -- 0x7ff35c0b3350 con 0x7ff338076500 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.695+0000 7ff352a00640 1 -- 172.21.15.163:0/256903634 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+680 (secure 0 0 0) 0x7ff35c0b3350 con 0x7ff338076500 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stdout: "atime": "2024-07-26 06:19:10", 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_pcent": "0.00", 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_quota": 10485760, 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stdout: "bytes_used": 0, 2024-07-26T06:19:13.695 INFO:teuthology.orchestra.run.smithi163.stdout: "created_at": "2024-07-26 06:19:10", 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "ctime": "2024-07-26 06:19:13", 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "data_pool": "cephfs_data", 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "features": [ 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot-clone", 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "snapshot-autoprotect" 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "flavor": 1, 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "gid": 0, 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "mode": 16877, 2024-07-26T06:19:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_addrs": [ 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6789", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.164:6789", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.164:6790" 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "mtime": "2024-07-26 06:19:10", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "path": "/volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_namespace": "", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "state": "complete", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "subvolume", 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout: "uid": 0 2024-07-26T06:19:13.697 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:13.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff338076500 msgr2=0x7ff3380789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff338076500 0x7ff3380789c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff34c036000 tx=0x7ff34c0046c0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 msgr2=0x7ff35c154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 0x7ff35c154d80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7ff35c161000 tx=0x7ff35400d970 comp rx=0 tx=0).stop 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 shutdown_connections 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff338076500 0x7ff3380789c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff35c154900 0x7ff35c154d80 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff35c160760 0x7ff35c153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 --2- 172.21.15.163:0/256903634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff340003a00 0x7ff35c15e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 >> 172.21.15.163:0/256903634 conn(0x7ff35c0d36b0 msgr2=0x7ff35c0cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.698+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 shutdown_connections 2024-07-26T06:19:13.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.699+0000 7ff361a00640 1 -- 172.21.15.163:0/256903634 wait complete. 2024-07-26T06:19:13.721 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol99070 --mode 777 --uid 1000 --gid 1000 2024-07-26T06:19:13.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7fe00640 1 -- 172.21.15.163:0/3659492531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c64004800 msgr2=0x7f7c64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/3659492531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c64004800 0x7f7c64004c80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c006f80 tx=0x7f7c64005830 comp rx=0 tx=0).stop 2024-07-26T06:19:13.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7f400640 1 -- 172.21.15.163:0/3659492531 reap_dead start 2024-07-26T06:19:13.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7fe00640 1 -- 172.21.15.163:0/3659492531 shutdown_connections 2024-07-26T06:19:13.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7fe00640 1 -- 172.21.15.163:0/3659492531 >> 172.21.15.163:0/3659492531 conn(0x7f7c8006d420 msgr2=0x7f7c8006d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:13.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7fe00640 1 -- 172.21.15.163:0/3659492531 shutdown_connections 2024-07-26T06:19:13.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.937+0000 7f7c7fe00640 1 -- 172.21.15.163:0/3659492531 wait complete. 2024-07-26T06:19:13.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.939+0000 7f7c7fe00640 1 Processor -- start 2024-07-26T06:19:13.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.939+0000 7f7c7fe00640 1 -- start start 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.940+0000 7f7c7fe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c64004800 0x7f7c8014edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.940+0000 7f7c7fe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c80150850 0x7f7c80152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.940+0000 7f7c7fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 0x7f7c80158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.940+0000 7f7c7fe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c800b4640 con 0x7f7c80153160 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.940+0000 7f7c7fe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7c80159aa0 con 0x7f7c64004800 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.940+0000 7f7c7fe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c800cca40 con 0x7f7c80150850 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 0x7f7c80158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 0x7f7c80158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45918/0 (socket says 172.21.15.163:45918) 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 -- 172.21.15.163:0/4009110310 learned_addr learned my addr 172.21.15.163:0/4009110310 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7e000640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c80150850 0x7f7c80152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7ea00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c64004800 0x7f7c8014edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 -- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c80150850 msgr2=0x7f7c80152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c80150850 0x7f7c80152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 -- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c64004800 msgr2=0x7f7c8014edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:13.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c64004800 0x7f7c8014edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:13.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 -- 172.21.15.163:0/4009110310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c80153650 con 0x7f7c80153160 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.941+0000 7f7c7f400640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 0x7f7c80158080 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7c800cefe0 tx=0x7f7c78010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.942+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800edb0 con 0x7f7c80153160 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.942+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c78015070 con 0x7f7c80153160 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.942+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c78003bd0 con 0x7f7c80153160 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.942+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c801f7440 con 0x7f7c80153160 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.942+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c801f7f20 con 0x7f7c80153160 2024-07-26T06:19:13.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.943+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c800cc500 con 0x7f7c80153160 2024-07-26T06:19:13.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.944+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f7c78004030 con 0x7f7c80153160 2024-07-26T06:19:13.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.946+0000 7f7c77400640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c60076480 0x7f7c60078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:13.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.946+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f7c780984e0 con 0x7f7c80153160 2024-07-26T06:19:13.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.946+0000 7f7c7ea00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c60076480 0x7f7c60078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:13.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.947+0000 7f7c7ea00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c60076480 0x7f7c60078940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7c70005fd0 tx=0x7f7c70005c50 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:13.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:13.950+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c78061b80 con 0x7f7c80153160 2024-07-26T06:19:14.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.191+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol99070", "mode": "777", "uid": 1000, "gid": 1000, "target": ["mon-mgr", ""]}) -- 0x7f7c801f81e0 con 0x7f7c60076480 2024-07-26T06:19:14.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.267+0000 7f7c77400640 1 -- 172.21.15.163:0/4009110310 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7c801f81e0 con 0x7f7c60076480 2024-07-26T06:19:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.270+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c60076480 msgr2=0x7f7c60078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.270+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c60076480 0x7f7c60078940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7c70005fd0 tx=0x7f7c70005c50 comp rx=0 tx=0).stop 2024-07-26T06:19:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 msgr2=0x7f7c80158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 0x7f7c80158080 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7c800cefe0 tx=0x7f7c78010e70 comp rx=0 tx=0).stop 2024-07-26T06:19:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 shutdown_connections 2024-07-26T06:19:14.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c60076480 0x7f7c60078940 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c80153160 0x7f7c80158080 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c80150850 0x7f7c80152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 --2- 172.21.15.163:0/4009110310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c64004800 0x7f7c8014edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 >> 172.21.15.163:0/4009110310 conn(0x7f7c8006d420 msgr2=0x7f7c8015c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:14.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 shutdown_connections 2024-07-26T06:19:14.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.271+0000 7f7c7fe00640 1 -- 172.21.15.163:0/4009110310 wait complete. 2024-07-26T06:19:14.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol99070 2024-07-26T06:19:14.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 -- 172.21.15.163:0/2039568547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 msgr2=0x7f7c18003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 --2- 172.21.15.163:0/2039568547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c18003e00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000baf0 tx=0x7f7c18005830 comp rx=0 tx=0).stop 2024-07-26T06:19:14.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 -- 172.21.15.163:0/2039568547 shutdown_connections 2024-07-26T06:19:14.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 --2- 172.21.15.163:0/2039568547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c18003e00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 -- 172.21.15.163:0/2039568547 >> 172.21.15.163:0/2039568547 conn(0x7f7c300b0e30 msgr2=0x7f7c300b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:14.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 -- 172.21.15.163:0/2039568547 shutdown_connections 2024-07-26T06:19:14.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.490+0000 7f7c37000640 1 -- 172.21.15.163:0/2039568547 wait complete. 2024-07-26T06:19:14.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.492+0000 7f7c37000640 1 Processor -- start 2024-07-26T06:19:14.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.492+0000 7f7c37000640 1 -- start start 2024-07-26T06:19:14.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c37000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c30141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c37000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c3015f9a0 0x7f7c30141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c37000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c3006d280 0x7f7c3006d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c37000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c300cbef0 con 0x7f7c18003a00 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c37000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30146df0 con 0x7f7c3006d280 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c37000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c300cc070 con 0x7f7c3015f9a0 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c35c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c30141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c35c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c30141980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:45932/0 (socket says 172.21.15.163:45932) 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.493+0000 7f7c35c00640 1 -- 172.21.15.163:0/1681763657 learned_addr learned my addr 172.21.15.163:0/1681763657 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c35c00640 1 -- 172.21.15.163:0/1681763657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c3015f9a0 msgr2=0x7f7c30141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c35c00640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c3015f9a0 0x7f7c30141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c35c00640 1 -- 172.21.15.163:0/1681763657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c3006d280 msgr2=0x7f7c3006d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c35c00640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c3006d280 0x7f7c3006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c35c00640 1 -- 172.21.15.163:0/1681763657 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c301eb670 con 0x7f7c18003a00 2024-07-26T06:19:14.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c35c00640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c30141980 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f7c20034040 tx=0x7f7c20031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:14.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20040070 con 0x7f7c18003a00 2024-07-26T06:19:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c20004230 con 0x7f7c18003a00 2024-07-26T06:19:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20035040 con 0x7f7c18003a00 2024-07-26T06:19:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.494+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c301eb0b0 con 0x7f7c18003a00 2024-07-26T06:19:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.495+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c30145870 con 0x7f7c18003a00 2024-07-26T06:19:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.496+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c300cc410 con 0x7f7c18003a00 2024-07-26T06:19:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.497+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f7c20038020 con 0x7f7c18003a00 2024-07-26T06:19:14.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.498+0000 7f7c2d600640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c140764b0 0x7f7c14078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:14.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.498+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f7c200b84a0 con 0x7f7c18003a00 2024-07-26T06:19:14.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.498+0000 7f7c35200640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c140764b0 0x7f7c14078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:14.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.499+0000 7f7c35200640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c140764b0 0x7f7c14078970 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7c240015c0 tx=0x7f7c24001930 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:14.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.502+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c2002c040 con 0x7f7c18003a00 2024-07-26T06:19:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.742+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol99070", "target": ["mon-mgr", ""]}) -- 0x7f7c300b25f0 con 0x7f7c140764b0 2024-07-26T06:19:14.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.820+0000 7f7c2d600640 1 -- 172.21.15.163:0/1681763657 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f7c300b25f0 con 0x7f7c140764b0 2024-07-26T06:19:14.820 INFO:teuthology.orchestra.run.smithi163.stdout:/volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b 2024-07-26T06:19:14.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c140764b0 msgr2=0x7f7c14078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c140764b0 0x7f7c14078970 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7c240015c0 tx=0x7f7c24001930 comp rx=0 tx=0).stop 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 msgr2=0x7f7c30141980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c30141980 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f7c20034040 tx=0x7f7c20031f90 comp rx=0 tx=0).stop 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 shutdown_connections 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7c140764b0 0x7f7c14078970 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c3006d280 0x7f7c3006d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c3015f9a0 0x7f7c30141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 --2- 172.21.15.163:0/1681763657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c18003a00 0x7f7c30141980 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 >> 172.21.15.163:0/1681763657 conn(0x7f7c300b0e30 msgr2=0x7f7c300736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 shutdown_connections 2024-07-26T06:19:14.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:14.823+0000 7f7c37000640 1 -- 172.21.15.163:0/1681763657 wait complete. 2024-07-26T06:19:14.844 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol99070 with 8 files each 1MB size under directory volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b 2024-07-26T06:19:14.844 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:14.844 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:14.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:14.892 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.031 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.032 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.032 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363471 s, 28.8 MB/s 2024-07-26T06:19:15.033 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.033 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:15.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:15.086 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.172 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.172 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.172 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392786 s, 26.7 MB/s 2024-07-26T06:19:15.174 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.174 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:15.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:15.229 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.318 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.318 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.318 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357678 s, 29.3 MB/s 2024-07-26T06:19:15.319 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.320 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:15.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:15.372 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.508 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.509 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.509 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369039 s, 28.4 MB/s 2024-07-26T06:19:15.510 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.511 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:15.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:15.564 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.642 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.642 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.642 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022503 s, 46.6 MB/s 2024-07-26T06:19:15.643 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.643 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:15.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:15.696 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.771 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.771 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.771 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268737 s, 39.0 MB/s 2024-07-26T06:19:15.773 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.773 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:15.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:15.825 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:15.952 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:15.952 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:15.952 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272973 s, 38.4 MB/s 2024-07-26T06:19:15.954 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:15.954 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:16.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:16.008 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol99070/e0a9306e-6ad0-4e72-94a1-c699d41d296b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2024-07-26T06:19:16.095 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:19:16.095 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:19:16.095 INFO:teuthology.orchestra.run.smithi163.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391735 s, 26.8 MB/s 2024-07-26T06:19:16.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol99070 subvol_snap15560 2024-07-26T06:19:16.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 -- 172.21.15.163:0/235637118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 msgr2=0x7f43a8071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:16.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 --2- 172.21.15.163:0/235637118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a8071530 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4390009830 tx=0x7f439002a5c0 comp rx=0 tx=0).stop 2024-07-26T06:19:16.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 -- 172.21.15.163:0/235637118 shutdown_connections 2024-07-26T06:19:16.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 --2- 172.21.15.163:0/235637118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a8071530 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:16.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 -- 172.21.15.163:0/235637118 >> 172.21.15.163:0/235637118 conn(0x7f43a80b0e30 msgr2=0x7f43a80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:16.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 -- 172.21.15.163:0/235637118 shutdown_connections 2024-07-26T06:19:16.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.311+0000 7f43a7e00640 1 -- 172.21.15.163:0/235637118 wait complete. 2024-07-26T06:19:16.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.313+0000 7f43a7e00640 1 Processor -- start 2024-07-26T06:19:16.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.313+0000 7f43a7e00640 1 -- start start 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f43a7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43a8071a70 0x7f43a8148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f43a7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a81f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f43a7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43a81556f0 0x7f43a814e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f43a7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43a80b41f0 con 0x7f43a80ce4a0 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f43a7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43a80b4070 con 0x7f43a8071a70 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f43a7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f43a80b4370 con 0x7f43a81556f0 2024-07-26T06:19:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f439fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a81f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:16.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f439fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a81f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37022/0 (socket says 172.21.15.163:37022) 2024-07-26T06:19:16.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.314+0000 7f439fe00640 1 -- 172.21.15.163:0/693832344 learned_addr learned my addr 172.21.15.163:0/693832344 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f43a6a00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43a8071a70 0x7f43a8148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439fe00640 1 -- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43a81556f0 msgr2=0x7f43a814e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f43a7400640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43a81556f0 0x7f43a814e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439fe00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43a81556f0 0x7f43a814e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439fe00640 1 -- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43a8071a70 msgr2=0x7f43a8148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439fe00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43a8071a70 0x7f43a8148cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439fe00640 1 -- 172.21.15.163:0/693832344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a81f2c90 con 0x7f43a80ce4a0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f43a6a00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43a8071a70 0x7f43a8148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439fe00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a81f22b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4390009950 tx=0x7f4390032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.315+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390040070 con 0x7f43a80ce4a0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.316+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4390005310 con 0x7f43a80ce4a0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.316+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43900047a0 con 0x7f43a80ce4a0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.316+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a81f2f70 con 0x7f43a80ce4a0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.316+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a8149c30 con 0x7f43a80ce4a0 2024-07-26T06:19:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.317+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a815e680 con 0x7f43a80ce4a0 2024-07-26T06:19:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.318+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f4390039040 con 0x7f43a80ce4a0 2024-07-26T06:19:16.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.319+0000 7f439ea00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4384076480 0x7f4384078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:16.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.319+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f43900b8c20 con 0x7f43a80ce4a0 2024-07-26T06:19:16.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.320+0000 7f43a6a00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4384076480 0x7f4384078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.320+0000 7f43a6a00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4384076480 0x7f4384078940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f43a81502d0 tx=0x7f4398009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:16.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.323+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43a815e680 con 0x7f43a80ce4a0 2024-07-26T06:19:16.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:16.566+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol99070", "snap_name": "subvol_snap15560", "target": ["mon-mgr", ""]}) -- 0x7f43a8071530 con 0x7f4384076480 2024-07-26T06:19:19.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.316+0000 7f439ea00640 1 -- 172.21.15.163:0/693832344 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f43a8071530 con 0x7f4384076480 2024-07-26T06:19:19.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4384076480 msgr2=0x7f4384078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4384076480 0x7f4384078940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f43a81502d0 tx=0x7f4398009040 comp rx=0 tx=0).stop 2024-07-26T06:19:19.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 msgr2=0x7f43a81f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a81f22b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4390009950 tx=0x7f4390032b90 comp rx=0 tx=0).stop 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 shutdown_connections 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4384076480 0x7f4384078940 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43a81556f0 0x7f43a814e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43a80ce4a0 0x7f43a81f22b0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 --2- 172.21.15.163:0/693832344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43a8071a70 0x7f43a8148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 >> 172.21.15.163:0/693832344 conn(0x7f43a80b0e30 msgr2=0x7f43a815d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.319+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 shutdown_connections 2024-07-26T06:19:19.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.320+0000 7f43a7e00640 1 -- 172.21.15.163:0/693832344 wait complete. 2024-07-26T06:19:19.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol99070 subvol_snap15560 subvol_clone33550 2024-07-26T06:19:19.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 -- 172.21.15.163:0/395969942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70900ce4a0 msgr2=0x7f7090071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 --2- 172.21.15.163:0/395969942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70900ce4a0 0x7f7090071530 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f707c009830 tx=0x7f707c02a510 comp rx=0 tx=0).stop 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 -- 172.21.15.163:0/395969942 shutdown_connections 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 --2- 172.21.15.163:0/395969942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7090071a70 0x7f70900b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 --2- 172.21.15.163:0/395969942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70900ce4a0 0x7f7090071530 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 -- 172.21.15.163:0/395969942 >> 172.21.15.163:0/395969942 conn(0x7f70900b0e30 msgr2=0x7f70900b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 -- 172.21.15.163:0/395969942 shutdown_connections 2024-07-26T06:19:19.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.556+0000 7f708f200640 1 -- 172.21.15.163:0/395969942 wait complete. 2024-07-26T06:19:19.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.558+0000 7f708f200640 1 Processor -- start 2024-07-26T06:19:19.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.558+0000 7f708f200640 1 -- start start 2024-07-26T06:19:19.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 0x7f7090143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:19.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70900ce4a0 0x7f70901e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:19.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70901556f0 0x7f70901e3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:19.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b3fe0 con 0x7f7090071a70 2024-07-26T06:19:19.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b3e60 con 0x7f70901556f0 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f70900b4160 con 0x7f70900ce4a0 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.559+0000 7f708de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 0x7f7090143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 0x7f7090143d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37046/0 (socket says 172.21.15.163:37046) 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 -- 172.21.15.163:0/3922232882 learned_addr learned my addr 172.21.15.163:0/3922232882 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 -- 172.21.15.163:0/3922232882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70900ce4a0 msgr2=0x7f70901e38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70900ce4a0 0x7f70901e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 -- 172.21.15.163:0/3922232882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70901556f0 msgr2=0x7f70901e3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70901556f0 0x7f70901e3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 -- 172.21.15.163:0/3922232882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7090144280 con 0x7f7090071a70 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f708de00640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 0x7f7090143d40 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f708000c3a0 tx=0x7f708000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:19.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.560+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080011790 con 0x7f7090071a70 2024-07-26T06:19:19.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.561+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7080017070 con 0x7f7090071a70 2024-07-26T06:19:19.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.561+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080011d20 con 0x7f7090071a70 2024-07-26T06:19:19.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.561+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70900ac4f0 con 0x7f7090071a70 2024-07-26T06:19:19.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.561+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70900acab0 con 0x7f7090071a70 2024-07-26T06:19:19.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.562+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70900b46f0 con 0x7f7090071a70 2024-07-26T06:19:19.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.563+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f708001c460 con 0x7f7090071a70 2024-07-26T06:19:19.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.564+0000 7f7086000640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70700764b0 0x7f7070078970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:19.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.565+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f70800978f0 con 0x7f7090071a70 2024-07-26T06:19:19.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.565+0000 7f708d400640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70700764b0 0x7f7070078970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:19.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.566+0000 7f708d400640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70700764b0 0x7f7070078970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f707c034000 tx=0x7f707c0046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:19.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.569+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7080060f90 con 0x7f7090071a70 2024-07-26T06:19:19.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.812+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol99070", "snap_name": "subvol_snap15560", "target_sub_name": "subvol_clone33550", "target": ["mon-mgr", ""]}) -- 0x7f70900ade60 con 0x7f70700764b0 2024-07-26T06:19:19.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.975+0000 7f7086000640 1 -- 172.21.15.163:0/3922232882 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f70900ade60 con 0x7f70700764b0 2024-07-26T06:19:19.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70700764b0 msgr2=0x7f7070078970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70700764b0 0x7f7070078970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f707c034000 tx=0x7f707c0046c0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 msgr2=0x7f7090143d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 0x7f7090143d40 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f708000c3a0 tx=0x7f708000c860 comp rx=0 tx=0).stop 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 shutdown_connections 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70700764b0 0x7f7070078970 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70901556f0 0x7f70901e3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70900ce4a0 0x7f70901e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 --2- 172.21.15.163:0/3922232882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7090071a70 0x7f7090143d40 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 >> 172.21.15.163:0/3922232882 conn(0x7f70900b0e30 msgr2=0x7f70900b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.978+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 shutdown_connections 2024-07-26T06:19:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:19.979+0000 7f708f200640 1 -- 172.21.15.163:0/3922232882 wait complete. 2024-07-26T06:19:19.984 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:19.983+0000 7fc4c7000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:19.984 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:19.983+0000 7fc4c7000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:19.984 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:19.983+0000 7fc4c7000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:19.984 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:19.983+0000 7fc4c7000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:19.984 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:19.983+0000 7fc4c7000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone33550 2024-07-26T06:19:20.007 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.005+0000 7fc4c6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.007 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.005+0000 7fc4c6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.007 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.005+0000 7fc4c6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.007 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.005+0000 7fc4c6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.007 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.005+0000 7fc4c6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 -- 172.21.15.163:0/2961383434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b00ce830 msgr2=0x7f99b00b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:20.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 --2- 172.21.15.163:0/2961383434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b00ce830 0x7f99b00b64c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f999c009830 tx=0x7f999c02a3f0 comp rx=0 tx=0).stop 2024-07-26T06:19:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 -- 172.21.15.163:0/2961383434 shutdown_connections 2024-07-26T06:19:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 --2- 172.21.15.163:0/2961383434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b00ce830 0x7f99b00b64c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 --2- 172.21.15.163:0/2961383434 >> v2:172.21.15.164:3300/0 conn(0x7f99b015d5a0 0x7f99b00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 -- 172.21.15.163:0/2961383434 >> 172.21.15.163:0/2961383434 conn(0x7f99b00cfda0 msgr2=0x7f99b00d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 -- 172.21.15.163:0/2961383434 shutdown_connections 2024-07-26T06:19:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.216+0000 7f99b5400640 1 -- 172.21.15.163:0/2961383434 wait complete. 2024-07-26T06:19:20.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.218+0000 7f99b5400640 1 Processor -- start 2024-07-26T06:19:20.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.218+0000 7f99b5400640 1 -- start start 2024-07-26T06:19:20.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.219+0000 7f99b5400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99b00b1e80 0x7f99b01f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:20.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.219+0000 7f99b5400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99b00ce830 0x7f99b0155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:20.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99b00b1e80 0x7f99b01f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99afe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99b00b1e80 0x7f99b01f37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37100/0 (socket says 172.21.15.163:37100) 2024-07-26T06:19:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99b5400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b015d5a0 0x7f99b0155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99b5400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f99b00b5130 con 0x7f99b00b1e80 2024-07-26T06:19:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99b5400640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f99b00b4fb0 con 0x7f99b00ce830 2024-07-26T06:19:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99b5400640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f99b00b52b0 con 0x7f99b015d5a0 2024-07-26T06:19:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99afe00640 1 -- 172.21.15.163:0/599992497 learned_addr learned my addr 172.21.15.163:0/599992497 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:20.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99af400640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99b00ce830 0x7f99b0155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99b4a00640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b015d5a0 0x7f99b0155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99afe00640 1 -- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b015d5a0 msgr2=0x7f99b0155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.220+0000 7f99afe00640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b015d5a0 0x7f99b0155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99afe00640 1 -- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99b00ce830 msgr2=0x7f99b0155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99afe00640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99b00ce830 0x7f99b0155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99afe00640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b0149a20 con 0x7f99b00b1e80 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99b4a00640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99b015d5a0 0x7f99b0155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99afe00640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99b00b1e80 0x7f99b01f37f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f99a000e790 tx=0x7f99a000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a000eea0 con 0x7f99b00b1e80 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99a000ccc0 con 0x7f99b00b1e80 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a0005280 con 0x7f99b00b1e80 2024-07-26T06:19:20.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.221+0000 7f99b5400640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b01583d0 con 0x7f99b00b1e80 2024-07-26T06:19:20.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.222+0000 7f99b5400640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b00c8000 con 0x7f99b00b1e80 2024-07-26T06:19:20.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.223+0000 7f99b5400640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b00ce2f0 con 0x7f99b00b1e80 2024-07-26T06:19:20.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.226+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 98354+0+0 (secure 0 0 0) 0x7f99a0005420 con 0x7f99b00b1e80 2024-07-26T06:19:20.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.227+0000 7f99a7e00640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9990076480 0x7f9990078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:20.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.227+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f99a0098850 con 0x7f99b00b1e80 2024-07-26T06:19:20.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.227+0000 7f99af400640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9990076480 0x7f9990078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:20.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.228+0000 7f99af400640 1 --2- 172.21.15.163:0/599992497 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9990076480 0x7f9990078940 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f999c034000 tx=0x7f999c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:20.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.230+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99a0061ec0 con 0x7f99b00b1e80 2024-07-26T06:19:20.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.434+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 19) ==== 98456+0+0 (secure 0 0 0) 0x7f99a005da50 con 0x7f99b00b1e80 2024-07-26T06:19:20.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.482+0000 7f99b5400640 1 -- 172.21.15.163:0/599992497 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone33550", "target": ["mon-mgr", ""]}) -- 0x7f99b00752b0 con 0x7f9990076480 2024-07-26T06:19:20.524 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.523+0000 7fc4c0c00640 -1 mgr.server reply reply (95) Operation not supported operation 'clone_internal' is not allowed on subvolume 'subvol99070' of type subvolume 2024-07-26T06:19:20.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:20.523+0000 7f99a7e00640 1 -- 172.21.15.163:0/599992497 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: -95 operation 'clone_internal' is not allowed on subvolume 'subvol99070' of type subvolume) ==== 94+0+0 (secure 0 0 0) 0x7f99b00752b0 con 0x7f9990076480 2024-07-26T06:19:20.525 INFO:teuthology.orchestra.run.smithi163.stderr:Error ENOTSUP: operation 'clone_internal' is not allowed on subvolume 'subvol99070' of type subvolume 2024-07-26T06:19:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 95 2024-07-26T06:19:20.540 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-07-26T06:19:20.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:19:20.650 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-07-26T06:19:20.777 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-07-26T06:19:20.777 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:19:20.777 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:20.777 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:20.835 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:20.835 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:20.887 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:19:20.986 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.985+0000 7fc4a9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.986 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.985+0000 7fc4a9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.986 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.985+0000 7fc4a9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.986 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.985+0000 7fc4a9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:20.986 INFO:tasks.ceph.mgr.y.smithi163.stderr:2024-07-26T06:19:20.985+0000 7fc4a9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:19:21.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a40e00640 1 -- 172.21.15.163:0/465066187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a20003a50 msgr2=0x7f1a20003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a40e00640 1 --2- 172.21.15.163:0/465066187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a20003a50 0x7f1a20003e70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00c290 tx=0x7f1a20005830 comp rx=0 tx=0).stop 2024-07-26T06:19:21.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a3a800640 1 -- 172.21.15.163:0/465066187 reap_dead start 2024-07-26T06:19:21.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a40e00640 1 -- 172.21.15.163:0/465066187 shutdown_connections 2024-07-26T06:19:21.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a40e00640 1 -- 172.21.15.163:0/465066187 >> 172.21.15.163:0/465066187 conn(0x7f1a3c06d420 msgr2=0x7f1a3c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:21.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a40e00640 1 -- 172.21.15.163:0/465066187 shutdown_connections 2024-07-26T06:19:21.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.101+0000 7f1a40e00640 1 -- 172.21.15.163:0/465066187 wait complete. 2024-07-26T06:19:21.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.103+0000 7f1a40e00640 1 Processor -- start 2024-07-26T06:19:21.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.103+0000 7f1a40e00640 1 -- start start 2024-07-26T06:19:21.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.104+0000 7f1a40e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a20003a50 0x7f1a3c0b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.104+0000 7f1a40e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c152630 0x7f1a3c1f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.104+0000 7f1a40e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 0x7f1a3c1472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.104+0000 7f1a40e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c15eea0 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.104+0000 7f1a40e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c15ed20 con 0x7f1a3c152630 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.104+0000 7f1a40e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3c15f020 con 0x7f1a20003a50 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 0x7f1a3c1472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 0x7f1a3c1472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37128/0 (socket says 172.21.15.163:37128) 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 -- 172.21.15.163:0/3347798019 learned_addr learned my addr 172.21.15.163:0/3347798019 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a39e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a20003a50 0x7f1a3c0b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a33e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c152630 0x7f1a3c1f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 -- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a20003a50 msgr2=0x7f1a3c0b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a20003a50 0x7f1a3c0b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 -- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c152630 msgr2=0x7f1a3c1f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c152630 0x7f1a3c1f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 -- 172.21.15.163:0/3347798019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c0b10e0 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.105+0000 7f1a3a800640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 0x7f1a3c1472a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1a340095f0 tx=0x7f1a34009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.106+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34016030 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.106+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a34004c10 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.106+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400e700 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.106+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3c152ab0 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.106+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3c0b0b20 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.107+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3c0b4220 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.108+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) ==== 98456+0+0 (secure 0 0 0) 0x7f1a34019070 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.109+0000 7f1a32000640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a180765d0 0x7f1a18078a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.109+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f1a34097ad0 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.109+0000 7f1a39e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a180765d0 0x7f1a18078a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.110+0000 7f1a39e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a180765d0 0x7f1a18078a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c033000 tx=0x7f1a2c02d000 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:21.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.113+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a34061020 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.337+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1a3c15cba0 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.338+0000 7f1a32000640 1 -- 172.21.15.163:0/3347798019 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v101) ==== 80+0+0 (secure 0 0 0) 0x7f1a34060920 con 0x7f1a3c1f76e0 2024-07-26T06:19:21.339 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:19:21.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a180765d0 msgr2=0x7f1a18078a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a180765d0 0x7f1a18078a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c033000 tx=0x7f1a2c02d000 comp rx=0 tx=0).stop 2024-07-26T06:19:21.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 msgr2=0x7f1a3c1472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 0x7f1a3c1472a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1a340095f0 tx=0x7f1a34009ab0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 shutdown_connections 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a180765d0 0x7f1a18078a90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3c1f76e0 0x7f1a3c1472a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c152630 0x7f1a3c1f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 --2- 172.21.15.163:0/3347798019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a20003a50 0x7f1a3c0b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 >> 172.21.15.163:0/3347798019 conn(0x7f1a3c06d420 msgr2=0x7f1a3c073650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.341+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 shutdown_connections 2024-07-26T06:19:21.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.342+0000 7f1a40e00640 1 -- 172.21.15.163:0/3347798019 wait complete. 2024-07-26T06:19:21.360 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-07-26T06:19:21.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:21.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-07-26T06:19:21.498 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:19:21.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:21.499 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-07-26T06:19:21.553 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:19:21.554 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:19:21.554 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:21.595 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:19:21.595 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:21.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-07-26T06:19:21.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 -- 172.21.15.163:0/3531878534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0cbad0 msgr2=0x7f381c0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 --2- 172.21.15.163:0/3531878534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0cbad0 0x7f381c0cd530 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f380c005fd0 tx=0x7f380c02a5d0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 -- 172.21.15.163:0/3531878534 shutdown_connections 2024-07-26T06:19:21.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 --2- 172.21.15.163:0/3531878534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f381c0cd000 0x7f381c0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 --2- 172.21.15.163:0/3531878534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0cbad0 0x7f381c0cd530 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 -- 172.21.15.163:0/3531878534 >> 172.21.15.163:0/3531878534 conn(0x7f381c0b0e30 msgr2=0x7f381c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:21.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 -- 172.21.15.163:0/3531878534 shutdown_connections 2024-07-26T06:19:21.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.883+0000 7f3822200640 1 -- 172.21.15.163:0/3531878534 wait complete. 2024-07-26T06:19:21.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.885+0000 7f3822200640 1 Processor -- start 2024-07-26T06:19:21.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.885+0000 7f3822200640 1 -- start start 2024-07-26T06:19:21.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3822200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 0x7f381c14cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3822200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f381c0cd000 0x7f381c15c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3822200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381c15c630 0x7f381c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3822200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f381c1594f0 con 0x7f381c0b4420 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3822200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f381c159370 con 0x7f381c15c630 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3822200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f381c159670 con 0x7f381c0cd000 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3820e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 0x7f381c14cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3820e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 0x7f381c14cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37150/0 (socket says 172.21.15.163:37150) 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3820e00640 1 -- 172.21.15.163:0/1174902181 learned_addr learned my addr 172.21.15.163:0/1174902181 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.886+0000 7f3813e00640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f381c0cd000 0x7f381c15c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3820e00640 1 -- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f381c0cd000 msgr2=0x7f381c15c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3820e00640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f381c0cd000 0x7f381c15c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3820e00640 1 -- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381c15c630 msgr2=0x7f381c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3820e00640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381c15c630 0x7f381c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3820e00640 1 -- 172.21.15.163:0/1174902181 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c15a990 con 0x7f381c0b4420 2024-07-26T06:19:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3820e00640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 0x7f381c14cc80 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f380c006010 tx=0x7f380c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:21.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c03f070 con 0x7f381c0b4420 2024-07-26T06:19:21.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f380c0047a0 con 0x7f381c0b4420 2024-07-26T06:19:21.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c035030 con 0x7f381c0b4420 2024-07-26T06:19:21.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c15ac70 con 0x7f381c0b4420 2024-07-26T06:19:21.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.887+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c15b230 con 0x7f381c0b4420 2024-07-26T06:19:21.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.888+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c0cd530 con 0x7f381c0b4420 2024-07-26T06:19:21.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.890+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) ==== 98456+0+0 (secure 0 0 0) 0x7f380c042070 con 0x7f381c0b4420 2024-07-26T06:19:21.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.891+0000 7f3819600640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f37f80765a0 0x7f37f8078a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:21.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.891+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f380c0b8bf0 con 0x7f381c0b4420 2024-07-26T06:19:21.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.891+0000 7f3813e00640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f37f80765a0 0x7f37f8078a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:21.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.892+0000 7f3813e00640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f37f80765a0 0x7f37f8078a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f381c072330 tx=0x7f3804009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:21.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:21.895+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f380c081f90 con 0x7f381c0b4420 2024-07-26T06:19:22.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.114+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f381c0734a0 con 0x7f381c0b4420 2024-07-26T06:19:22.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.115+0000 7f3819600640 1 -- 172.21.15.163:0/1174902181 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v101) ==== 80+0+0 (secure 0 0 0) 0x7f380c081890 con 0x7f381c0b4420 2024-07-26T06:19:22.116 INFO:teuthology.orchestra.run.smithi163.stderr:listed 0 entries 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f37f80765a0 msgr2=0x7f37f8078a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f37f80765a0 0x7f37f8078a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f381c072330 tx=0x7f3804009040 comp rx=0 tx=0).stop 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 msgr2=0x7f381c14cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 0x7f381c14cc80 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f380c006010 tx=0x7f380c0038a0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 shutdown_connections 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f37f80765a0 0x7f37f8078a60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381c15c630 0x7f381c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f381c0cd000 0x7f381c15c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 --2- 172.21.15.163:0/1174902181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f381c0b4420 0x7f381c14cc80 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 >> 172.21.15.163:0/1174902181 conn(0x7f381c0b0e30 msgr2=0x7f381c146280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:22.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 shutdown_connections 2024-07-26T06:19:22.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.118+0000 7f3822200640 1 -- 172.21.15.163:0/1174902181 wait complete. 2024-07-26T06:19:22.136 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-07-26T06:19:22.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:22.136 DEBUG:teuthology.orchestra.run.smithi164:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-07-26T06:19:22.243 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:19:22.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:22.243 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-07-26T06:19:22.289 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:19:22.289 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:22.289 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:22.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:19:22.347 INFO:tasks.cephfs.mount:Terminating background process 2024-07-26T06:19:22.348 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:19:22.348 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:22.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:19:22.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:19:22.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 -- 172.21.15.163:0/2221039116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c146be0 msgr2=0x7fc30c0ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 --2- 172.21.15.163:0/2221039116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c146be0 0x7fc30c0ad8e0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc30c15e970 tx=0x7fc2fc02a540 comp rx=0 tx=0).stop 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 -- 172.21.15.163:0/2221039116 shutdown_connections 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 --2- 172.21.15.163:0/2221039116 >> v2:172.21.15.164:3300/0 conn(0x7fc30c0ade20 0x7fc30c0cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 --2- 172.21.15.163:0/2221039116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c146be0 0x7fc30c0ad8e0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 -- 172.21.15.163:0/2221039116 >> 172.21.15.163:0/2221039116 conn(0x7fc30c0b0e30 msgr2=0x7fc30c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 -- 172.21.15.163:0/2221039116 shutdown_connections 2024-07-26T06:19:22.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.608+0000 7fc30b200640 1 -- 172.21.15.163:0/2221039116 wait complete. 2024-07-26T06:19:22.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.610+0000 7fc30b200640 1 Processor -- start 2024-07-26T06:19:22.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.610+0000 7fc30b200640 1 -- start start 2024-07-26T06:19:22.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.611+0000 7fc30b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 0x7fc30c13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:22.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.611+0000 7fc30b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30c0cc280 0x7fc30c13ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:22.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.611+0000 7fc30b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc30c146be0 0x7fc30c1446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:22.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.611+0000 7fc30b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0b48e0 con 0x7fc30c0ade20 2024-07-26T06:19:22.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.611+0000 7fc30b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0b4760 con 0x7fc30c146be0 2024-07-26T06:19:22.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.611+0000 7fc30b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c0ce520 con 0x7fc30c0cc280 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 0x7fc30c13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 0x7fc30c13fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37180/0 (socket says 172.21.15.163:37180) 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 -- 172.21.15.163:0/1162418371 learned_addr learned my addr 172.21.15.163:0/1162418371 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc301400640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30c0cc280 0x7fc30c13ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 -- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30c0cc280 msgr2=0x7fc30c13ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc30a800640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc30c146be0 0x7fc30c1446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30c0cc280 0x7fc30c13ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 -- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc30c146be0 msgr2=0x7fc30c1446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc30c146be0 0x7fc30c1446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 -- 172.21.15.163:0/1162418371 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c143610 con 0x7fc30c0ade20 2024-07-26T06:19:22.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc30a800640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc30c146be0 0x7fc30c1446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:22.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.612+0000 7fc309e00640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 0x7fc30c13fd90 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009780 tx=0x7fc2fc033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.613+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc031d60 con 0x7fc30c0ade20 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.613+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2fc005020 con 0x7fc30c0ade20 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.613+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc005470 con 0x7fc30c0ade20 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.613+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c1438f0 con 0x7fc30c0ade20 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.613+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c143eb0 con 0x7fc30c0ade20 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.614+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c0cea20 con 0x7fc30c0ade20 2024-07-26T06:19:22.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.615+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fc2fc0047a0 con 0x7fc30c0ade20 2024-07-26T06:19:22.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.616+0000 7fc302a00640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc2e8076630 0x7fc2e8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:22.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.616+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7fc2fc0b9030 con 0x7fc30c0ade20 2024-07-26T06:19:22.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.617+0000 7fc301400640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc2e8076630 0x7fc2e8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:22.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.617+0000 7fc301400640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc2e8076630 0x7fc2e8078af0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc30c1465f0 tx=0x7fc2f4005c80 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:22.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.621+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2fc03f030 con 0x7fc30c0ade20 2024-07-26T06:19:22.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.867+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc30c0b5170 con 0x7fc30c0ade20 2024-07-26T06:19:22.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.868+0000 7fc302a00640 1 -- 172.21.15.163:0/1162418371 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5234 (secure 0 0 0) 0x7fc2fc082330 con 0x7fc30c0ade20 2024-07-26T06:19:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:22.869 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":71,"btime":"2024-07-26T06:19:09:653692+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:58.690202+0000","modified":"2024-07-26T06:19:01.753809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6834/1667488712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1667488712},{"type":"v1","addr":"172.21.15.163:6834","nonce":1667488712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":8}]} 2024-07-26T06:19:22.870 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 71 2024-07-26T06:19:22.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc2e8076630 msgr2=0x7fc2e8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc2e8076630 0x7fc2e8078af0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc30c1465f0 tx=0x7fc2f4005c80 comp rx=0 tx=0).stop 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 msgr2=0x7fc30c13fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 0x7fc30c13fd90 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009780 tx=0x7fc2fc033d60 comp rx=0 tx=0).stop 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 shutdown_connections 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fc2e8076630 0x7fc2e8078af0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc30c146be0 0x7fc30c1446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30c0cc280 0x7fc30c13ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 --2- 172.21.15.163:0/1162418371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc30c0ade20 0x7fc30c13fd90 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:22.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 >> 172.21.15.163:0/1162418371 conn(0x7fc30c0b0e30 msgr2=0x7fc30c0b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:22.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 shutdown_connections 2024-07-26T06:19:22.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:22.871+0000 7fc30b200640 1 -- 172.21.15.163:0/1162418371 wait complete. 2024-07-26T06:19:22.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7e00640 1 -- 172.21.15.163:0/4161337815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9ac004800 msgr2=0x7ff9ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/4161337815 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9ac004800 0x7ff9ac004c80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c2e0 tx=0x7ff9ac005830 comp rx=0 tx=0).stop 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7400640 1 -- 172.21.15.163:0/4161337815 reap_dead start 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7e00640 1 -- 172.21.15.163:0/4161337815 shutdown_connections 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7e00640 1 -- 172.21.15.163:0/4161337815 >> 172.21.15.163:0/4161337815 conn(0x7ff9c80b0e30 msgr2=0x7ff9c80b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7e00640 1 -- 172.21.15.163:0/4161337815 shutdown_connections 2024-07-26T06:19:23.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.104+0000 7ff9c7e00640 1 -- 172.21.15.163:0/4161337815 wait complete. 2024-07-26T06:19:23.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.106+0000 7ff9c7e00640 1 Processor -- start 2024-07-26T06:19:23.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.106+0000 7ff9c7e00640 1 -- start start 2024-07-26T06:19:23.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9ac004800 0x7ff9c81399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 0x7ff9c80cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c80add90 0x7ff9c80abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c8164310 con 0x7ff9c80cd470 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c8164190 con 0x7ff9ac004800 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c8164490 con 0x7ff9c80add90 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 0x7ff9c80cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 0x7ff9c80cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37196/0 (socket says 172.21.15.163:37196) 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 -- 172.21.15.163:0/378415233 learned_addr learned my addr 172.21.15.163:0/378415233 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6a00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9ac004800 0x7ff9c81399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c7400640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c80add90 0x7ff9c80abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 -- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c80add90 msgr2=0x7ff9c80abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c80add90 0x7ff9c80abcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 -- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9ac004800 msgr2=0x7ff9c81399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9ac004800 0x7ff9c81399a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.107+0000 7ff9c6000640 1 -- 172.21.15.163:0/378415233 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c80cc3d0 con 0x7ff9c80cd470 2024-07-26T06:19:23.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.108+0000 7ff9c6000640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 0x7ff9c80cbb80 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c340 tx=0x7ff9b4006db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:23.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.108+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4003a20 con 0x7ff9c80cd470 2024-07-26T06:19:23.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.108+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9b4003cb0 con 0x7ff9c80cd470 2024-07-26T06:19:23.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.108+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c80cc6b0 con 0x7ff9c80cd470 2024-07-26T06:19:23.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.108+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b402e6a0 con 0x7ff9c80cd470 2024-07-26T06:19:23.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.108+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c80ccc70 con 0x7ff9c80cd470 2024-07-26T06:19:23.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.110+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c8071530 con 0x7ff9c80cd470 2024-07-26T06:19:23.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.111+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7ff9b4035020 con 0x7ff9c80cd470 2024-07-26T06:19:23.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.112+0000 7ff9bf400640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9a8076660 0x7ff9a8078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.112+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7ff9b40b8030 con 0x7ff9c80cd470 2024-07-26T06:19:23.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.112+0000 7ff9c6a00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9a8076660 0x7ff9a8078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.113+0000 7ff9c6a00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9a8076660 0x7ff9a8078b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8001690 tx=0x7ff9b8009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:23.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.116+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff9b4047030 con 0x7ff9c80cd470 2024-07-26T06:19:23.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.369+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9c815d970 con 0x7ff9c80cd470 2024-07-26T06:19:23.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.370+0000 7ff9bf400640 1 -- 172.21.15.163:0/378415233 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5234 (secure 0 0 0) 0x7ff9b4081430 con 0x7ff9c80cd470 2024-07-26T06:19:23.371 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:23.371 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":71,"btime":"2024-07-26T06:19:09:653692+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:58.690202+0000","modified":"2024-07-26T06:19:01.753809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6834/1667488712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1667488712},{"type":"v1","addr":"172.21.15.163:6834","nonce":1667488712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":8}]} 2024-07-26T06:19:23.372 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 71 2024-07-26T06:19:23.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9a8076660 msgr2=0x7ff9a8078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9a8076660 0x7ff9a8078b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8001690 tx=0x7ff9b8009040 comp rx=0 tx=0).stop 2024-07-26T06:19:23.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 msgr2=0x7ff9c80cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 0x7ff9c80cbb80 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400c340 tx=0x7ff9b4006db0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 shutdown_connections 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff9a8076660 0x7ff9a8078b20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c80add90 0x7ff9c80abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c80cd470 0x7ff9c80cbb80 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 --2- 172.21.15.163:0/378415233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9ac004800 0x7ff9c81399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 >> 172.21.15.163:0/378415233 conn(0x7ff9c80b0e30 msgr2=0x7ff9c814c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 shutdown_connections 2024-07-26T06:19:23.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.373+0000 7ff9c7e00640 1 -- 172.21.15.163:0/378415233 wait complete. 2024-07-26T06:19:23.394 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:19:23.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 -- 172.21.15.163:0/1720598448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 msgr2=0x7feacc0b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 --2- 172.21.15.163:0/1720598448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc0b6a00 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7feac4005fd0 tx=0x7feac402a4a0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 -- 172.21.15.163:0/1720598448 shutdown_connections 2024-07-26T06:19:23.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 --2- 172.21.15.163:0/1720598448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc0b6a00 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 --2- 172.21.15.163:0/1720598448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feacc0b1e80 0x7feacc0b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 -- 172.21.15.163:0/1720598448 >> 172.21.15.163:0/1720598448 conn(0x7feacc0cfda0 msgr2=0x7feacc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:23.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 -- 172.21.15.163:0/1720598448 shutdown_connections 2024-07-26T06:19:23.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.603+0000 7feacbe00640 1 -- 172.21.15.163:0/1720598448 wait complete. 2024-07-26T06:19:23.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.605+0000 7feacbe00640 1 Processor -- start 2024-07-26T06:19:23.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.605+0000 7feacbe00640 1 -- start start 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feacbe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feacc0b1e80 0x7feacc1f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feacbe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feacbe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feacc15d5a0 0x7feacc0c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feacbe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7feacc0b4f20 con 0x7feacc0b23d0 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feacbe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feacc0b4da0 con 0x7feacc0b1e80 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feacbe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feacc0b50a0 con 0x7feacc15d5a0 2024-07-26T06:19:23.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feaca000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feaca000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37214/0 (socket says 172.21.15.163:37214) 2024-07-26T06:19:23.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.606+0000 7feaca000640 1 -- 172.21.15.163:0/230328870 learned_addr learned my addr 172.21.15.163:0/230328870 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feacaa00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feacc0b1e80 0x7feacc1f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feacb400640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feacc15d5a0 0x7feacc0c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feaca000640 1 -- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feacc15d5a0 msgr2=0x7feacc0c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feaca000640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feacc15d5a0 0x7feacc0c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feaca000640 1 -- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feacc0b1e80 msgr2=0x7feacc1f36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feaca000640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feacc0b1e80 0x7feacc1f36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feaca000640 1 -- 172.21.15.163:0/230328870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc1564e0 con 0x7feacc0b23d0 2024-07-26T06:19:23.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feacaa00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feacc0b1e80 0x7feacc1f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:23.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.607+0000 7feaca000640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc155e90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7feab400c970 tx=0x7feab400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:23.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.608+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4004220 con 0x7feacc0b23d0 2024-07-26T06:19:23.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.608+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feab4007950 con 0x7feacc0b23d0 2024-07-26T06:19:23.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.608+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4007da0 con 0x7feacc0b23d0 2024-07-26T06:19:23.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.608+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feacc1567c0 con 0x7feacc0b23d0 2024-07-26T06:19:23.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.608+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feacc156d80 con 0x7feacc0b23d0 2024-07-26T06:19:23.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.609+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feacc0b5590 con 0x7feacc0b23d0 2024-07-26T06:19:23.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.610+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7feab4003610 con 0x7feacc0b23d0 2024-07-26T06:19:23.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.611+0000 7feac2a00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7feaa8076630 0x7feaa8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:23.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.611+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7feab4098010 con 0x7feacc0b23d0 2024-07-26T06:19:23.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.611+0000 7feacaa00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7feaa8076630 0x7feaa8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:23.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.612+0000 7feacaa00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7feaa8076630 0x7feaa8078af0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feabc007e90 tx=0x7feabc0073a0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:23.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.615+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feab4014030 con 0x7feacc0b23d0 2024-07-26T06:19:23.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.831+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feacc0c8d80 con 0x7feacc0b23d0 2024-07-26T06:19:23.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.835+0000 7feac2a00640 1 -- 172.21.15.163:0/230328870 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+16735 (secure 0 0 0) 0x7feab40612d0 con 0x7feacc0b23d0 2024-07-26T06:19:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:23.836 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":101,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:19:06.569948+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:56.563057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:57.693372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:19:23.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7feaa8076630 msgr2=0x7feaa8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7feaa8076630 0x7feaa8078af0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feabc007e90 tx=0x7feabc0073a0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 msgr2=0x7feacc155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:23.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc155e90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7feab400c970 tx=0x7feab400ce30 comp rx=0 tx=0).stop 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 shutdown_connections 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7feaa8076630 0x7feaa8078af0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feacc15d5a0 0x7feacc0c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feacc0b23d0 0x7feacc155e90 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 --2- 172.21.15.163:0/230328870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feacc0b1e80 0x7feacc1f36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 >> 172.21.15.163:0/230328870 conn(0x7feacc0cfda0 msgr2=0x7feacc15c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 shutdown_connections 2024-07-26T06:19:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:23.838+0000 7feacbe00640 1 -- 172.21.15.163:0/230328870 wait complete. 2024-07-26T06:19:23.860 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:19:23.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:19:24.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 -- 172.21.15.163:0/2415774226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7180b1e80 msgr2=0x7fd7180b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 --2- 172.21.15.163:0/2415774226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7180b1e80 0x7fd7180b6a00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fd7180690b0 tx=0x7fd71002a8f0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 -- 172.21.15.163:0/2415774226 shutdown_connections 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 --2- 172.21.15.163:0/2415774226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd7180b1e80 0x7fd7180b6a00 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 --2- 172.21.15.163:0/2415774226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180ce830 0x7fd7180b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 --2- 172.21.15.163:0/2415774226 >> v2:172.21.15.164:3300/0 conn(0x7fd71815d5a0 0x7fd7180ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 -- 172.21.15.163:0/2415774226 >> 172.21.15.163:0/2415774226 conn(0x7fd7180cfda0 msgr2=0x7fd7180d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 -- 172.21.15.163:0/2415774226 shutdown_connections 2024-07-26T06:19:24.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.067+0000 7fd71e400640 1 -- 172.21.15.163:0/2415774226 wait complete. 2024-07-26T06:19:24.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.069+0000 7fd71e400640 1 Processor -- start 2024-07-26T06:19:24.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.069+0000 7fd71e400640 1 -- start start 2024-07-26T06:19:24.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71e400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180b1e80 0x7fd718156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71e400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180ce830 0x7fd7180c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71e400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 0x7fd71815e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71e400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd7180b4da0 con 0x7fd71815d5a0 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71e400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd7180b2660 con 0x7fd7180b1e80 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71e400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd7180b4f20 con 0x7fd7180ce830 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 0x7fd71815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 0x7fd71815e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37230/0 (socket says 172.21.15.163:37230) 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 -- 172.21.15.163:0/1784846244 learned_addr learned my addr 172.21.15.163:0/1784846244 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71d000640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180b1e80 0x7fd718156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd717e00640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180ce830 0x7fd7180c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 -- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180ce830 msgr2=0x7fd7180c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180ce830 0x7fd7180c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 -- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180b1e80 msgr2=0x7fd718156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180b1e80 0x7fd718156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.070+0000 7fd71da00640 1 -- 172.21.15.163:0/1784846244 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7181f43e0 con 0x7fd71815d5a0 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.071+0000 7fd71da00640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 0x7fd71815e270 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd710009d60 tx=0x7fd710003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.071+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710004040 con 0x7fd71815d5a0 2024-07-26T06:19:24.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.071+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd71002d070 con 0x7fd71815d5a0 2024-07-26T06:19:24.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.071+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710004440 con 0x7fd71815d5a0 2024-07-26T06:19:24.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.071+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7181f46c0 con 0x7fd71815d5a0 2024-07-26T06:19:24.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.071+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7180b79d0 con 0x7fd71815d5a0 2024-07-26T06:19:24.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.073+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7180ce2f0 con 0x7fd71815d5a0 2024-07-26T06:19:24.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.074+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fd7100045e0 con 0x7fd71815d5a0 2024-07-26T06:19:24.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.074+0000 7fd714c00640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd6f8076660 0x7fd6f8078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.075+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7fd7100b9810 con 0x7fd71815d5a0 2024-07-26T06:19:24.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.075+0000 7fd71d000640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd6f8076660 0x7fd6f8078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.075+0000 7fd71d000640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd6f8076660 0x7fd6f8078b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd7080058c0 tx=0x7fd70800c440 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:24.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.080+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd710082d20 con 0x7fd71815d5a0 2024-07-26T06:19:24.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.337+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd718158270 con 0x7fd71815d5a0 2024-07-26T06:19:24.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.337+0000 7fd714c00640 1 -- 172.21.15.163:0/1784846244 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5234 (secure 0 0 0) 0x7fd710082620 con 0x7fd71815d5a0 2024-07-26T06:19:24.338 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:24.339 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":71,"btime":"2024-07-26T06:19:09:653692+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:58.690202+0000","modified":"2024-07-26T06:19:01.753809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6834/1667488712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1667488712},{"type":"v1","addr":"172.21.15.163:6834","nonce":1667488712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":8}]} 2024-07-26T06:19:24.339 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 71 2024-07-26T06:19:24.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd6f8076660 msgr2=0x7fd6f8078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd6f8076660 0x7fd6f8078b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd7080058c0 tx=0x7fd70800c440 comp rx=0 tx=0).stop 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 msgr2=0x7fd71815e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 0x7fd71815e270 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd710009d60 tx=0x7fd710003c10 comp rx=0 tx=0).stop 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 shutdown_connections 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fd6f8076660 0x7fd6f8078b20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd71815d5a0 0x7fd71815e270 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180ce830 0x7fd7180c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 --2- 172.21.15.163:0/1784846244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180b1e80 0x7fd718156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 >> 172.21.15.163:0/1784846244 conn(0x7fd7180cfda0 msgr2=0x7fd7180d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 shutdown_connections 2024-07-26T06:19:24.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.341+0000 7fd71e400640 1 -- 172.21.15.163:0/1784846244 wait complete. 2024-07-26T06:19:24.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.567+0000 7fb829a00640 1 -- 172.21.15.163:0/34849517 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8240cb7e0 msgr2=0x7fb82415ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.567+0000 7fb829a00640 1 --2- 172.21.15.163:0/34849517 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8240cb7e0 0x7fb82415ed20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb82406f660 tx=0x7fb81c02a9a0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.567+0000 7fb829a00640 1 -- 172.21.15.163:0/34849517 shutdown_connections 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.567+0000 7fb829a00640 1 --2- 172.21.15.163:0/34849517 >> v2:172.21.15.164:3300/0 conn(0x7fb824071cf0 0x7fb82415be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.567+0000 7fb829a00640 1 --2- 172.21.15.163:0/34849517 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb8240cb7e0 0x7fb82415ed20 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.567+0000 7fb829a00640 1 -- 172.21.15.163:0/34849517 >> 172.21.15.163:0/34849517 conn(0x7fb82406d420 msgr2=0x7fb82406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.568+0000 7fb829a00640 1 -- 172.21.15.163:0/34849517 shutdown_connections 2024-07-26T06:19:24.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.568+0000 7fb829a00640 1 -- 172.21.15.163:0/34849517 wait complete. 2024-07-26T06:19:24.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.569+0000 7fb829a00640 1 Processor -- start 2024-07-26T06:19:24.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.569+0000 7fb829a00640 1 -- start start 2024-07-26T06:19:24.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb829a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 0x7fb8240add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb829a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8240cb7e0 0x7fb8241f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb829a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82415c380 0x7fb8240b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb829a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb824073530 con 0x7fb824071cf0 2024-07-26T06:19:24.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb829a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb82406c860 con 0x7fb8240cb7e0 2024-07-26T06:19:24.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb829a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb8240736b0 con 0x7fb82415c380 2024-07-26T06:19:24.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.570+0000 7fb822a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 0x7fb8240add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 0x7fb8240add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:37250/0 (socket says 172.21.15.163:37250) 2024-07-26T06:19:24.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 -- 172.21.15.163:0/1223133786 learned_addr learned my addr 172.21.15.163:0/1223133786 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 -- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82415c380 msgr2=0x7fb8240b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82415c380 0x7fb8240b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 -- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8240cb7e0 msgr2=0x7fb8241f1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822000640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8240cb7e0 0x7fb8241f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8240cb7e0 0x7fb8241f1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 -- 172.21.15.163:0/1223133786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8241f1d00 con 0x7fb824071cf0 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822000640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8240cb7e0 0x7fb8241f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.571+0000 7fb822a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 0x7fb8240add50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009bf0 tx=0x7fb81c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:24.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.572+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c047070 con 0x7fb824071cf0 2024-07-26T06:19:24.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.572+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb81c004250 con 0x7fb824071cf0 2024-07-26T06:19:24.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.572+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c033050 con 0x7fb824071cf0 2024-07-26T06:19:24.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.572+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8241f1fe0 con 0x7fb824071cf0 2024-07-26T06:19:24.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.572+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb82413d430 con 0x7fb824071cf0 2024-07-26T06:19:24.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.573+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb82415be40 con 0x7fb824071cf0 2024-07-26T06:19:24.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.574+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fb81c04a070 con 0x7fb824071cf0 2024-07-26T06:19:24.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.576+0000 7fb81aa00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb800076630 0x7fb800078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:24.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.576+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7fb81c0b8e00 con 0x7fb824071cf0 2024-07-26T06:19:24.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.576+0000 7fb822000640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb800076630 0x7fb800078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:24.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.577+0000 7fb822000640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb800076630 0x7fb800078af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb80c006fd0 tx=0x7fb80c008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:24.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.580+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb81c082290 con 0x7fb824071cf0 2024-07-26T06:19:24.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.826+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8240ae300 con 0x7fb824071cf0 2024-07-26T06:19:24.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.827+0000 7fb81aa00640 1 -- 172.21.15.163:0/1223133786 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5234 (secure 0 0 0) 0x7fb81c081b90 con 0x7fb824071cf0 2024-07-26T06:19:24.828 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:24.828 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":71,"btime":"2024-07-26T06:19:09:653692+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T06:18:58.690202+0000","modified":"2024-07-26T06:19:01.753809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.163:6834/1667488712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":1667488712},{"type":"v1","addr":"172.21.15.163:6834","nonce":1667488712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":8}]} 2024-07-26T06:19:24.829 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 71 2024-07-26T06:19:24.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb800076630 msgr2=0x7fb800078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb800076630 0x7fb800078af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb80c006fd0 tx=0x7fb80c008040 comp rx=0 tx=0).stop 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 msgr2=0x7fb8240add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 0x7fb8240add50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009bf0 tx=0x7fb81c003f10 comp rx=0 tx=0).stop 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 shutdown_connections 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb800076630 0x7fb800078af0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82415c380 0x7fb8240b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8240cb7e0 0x7fb8241f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 --2- 172.21.15.163:0/1223133786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb824071cf0 0x7fb8240add50 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.830+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 >> 172.21.15.163:0/1223133786 conn(0x7fb82406d420 msgr2=0x7fb8240b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.831+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 shutdown_connections 2024-07-26T06:19:24.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:24.831+0000 7fb829a00640 1 -- 172.21.15.163:0/1223133786 wait complete. 2024-07-26T06:19:24.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:19:25.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 -- 172.21.15.163:0/2489961065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c0b1e80 msgr2=0x7fc01c0b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 --2- 172.21.15.163:0/2489961065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0b6bd0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0690b0 tx=0x7fc01402a980 comp rx=0 tx=0).stop 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 -- 172.21.15.163:0/2489961065 shutdown_connections 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 --2- 172.21.15.163:0/2489961065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0b6bd0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 --2- 172.21.15.163:0/2489961065 >> v2:172.21.15.164:3300/0 conn(0x7fc01c15d5a0 0x7fc01c0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 -- 172.21.15.163:0/2489961065 >> 172.21.15.163:0/2489961065 conn(0x7fc01c0cfda0 msgr2=0x7fc01c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 -- 172.21.15.163:0/2489961065 shutdown_connections 2024-07-26T06:19:25.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.053+0000 7fc01b200640 1 -- 172.21.15.163:0/2489961065 wait complete. 2024-07-26T06:19:25.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.055+0000 7fc01b200640 1 Processor -- start 2024-07-26T06:19:25.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.055+0000 7fc01b200640 1 -- start start 2024-07-26T06:19:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01b200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01b200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01c0ce830 0x7fc01c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01b200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 0x7fc01c1413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01b200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c0b5130 con 0x7fc01c15d5a0 2024-07-26T06:19:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01b200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c0b4fb0 con 0x7fc01c0b1e80 2024-07-26T06:19:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01b200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c0b52b0 con 0x7fc01c0ce830 2024-07-26T06:19:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.056+0000 7fc01a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 0x7fc01c1413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 0x7fc01c1413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55900/0 (socket says 172.21.15.163:55900) 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 -- 172.21.15.163:0/870098942 learned_addr learned my addr 172.21.15.163:0/870098942 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc019e00640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 -- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01c0ce830 msgr2=0x7fc01c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01c0ce830 0x7fc01c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 -- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01c0b1e80 msgr2=0x7fc01c0ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 -- 172.21.15.163:0/870098942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c0c7f40 con 0x7fc01c15d5a0 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc019e00640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.057+0000 7fc01a800640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 0x7fc01c1413b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc01402af70 tx=0x7fc014003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.058+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014034050 con 0x7fc01c15d5a0 2024-07-26T06:19:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.058+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc014004450 con 0x7fc01c15d5a0 2024-07-26T06:19:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.058+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014043440 con 0x7fc01c15d5a0 2024-07-26T06:19:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.058+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01c0c8220 con 0x7fc01c15d5a0 2024-07-26T06:19:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.058+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c0c87e0 con 0x7fc01c15d5a0 2024-07-26T06:19:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.059+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc01c0ca0b0 con 0x7fc01c15d5a0 2024-07-26T06:19:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.060+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fc014037020 con 0x7fc01c15d5a0 2024-07-26T06:19:25.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.061+0000 7fc012000640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbffc076630 0x7fbffc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.061+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7fc0140b9230 con 0x7fc01c15d5a0 2024-07-26T06:19:25.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.061+0000 7fc019e00640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbffc076630 0x7fbffc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.062+0000 7fc019e00640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbffc076630 0x7fbffc078af0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009aa0 tx=0x7fc00c005db0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.065+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc01c0ca0b0 con 0x7fc01c15d5a0 2024-07-26T06:19:25.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.277+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc01c074680 con 0x7fc01c15d5a0 2024-07-26T06:19:25.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.280+0000 7fc012000640 1 -- 172.21.15.163:0/870098942 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+16735 (secure 0 0 0) 0x7fc014082640 con 0x7fc01c15d5a0 2024-07-26T06:19:25.281 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:25.281 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":101,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:19:06.569948+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:18:56.563057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-07-26T06:18:57.693372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.282+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbffc076630 msgr2=0x7fbffc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.282+0000 7fc01b200640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbffc076630 0x7fbffc078af0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009aa0 tx=0x7fc00c005db0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 msgr2=0x7fc01c1413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 0x7fc01c1413b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc01402af70 tx=0x7fc014003f10 comp rx=0 tx=0).stop 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 shutdown_connections 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbffc076630 0x7fbffc078af0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc01c15d5a0 0x7fc01c1413b0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01c0ce830 0x7fc01c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 --2- 172.21.15.163:0/870098942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01c0b1e80 0x7fc01c0ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 >> 172.21.15.163:0/870098942 conn(0x7fc01c0cfda0 msgr2=0x7fc01c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:25.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 shutdown_connections 2024-07-26T06:19:25.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.283+0000 7fc01b200640 1 -- 172.21.15.163:0/870098942 wait complete. 2024-07-26T06:19:25.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-07-26T06:19:25.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 -- 172.21.15.163:0/526548988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 msgr2=0x7f99fc06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 --2- 172.21.15.163:0/526548988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc06bf40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f99fc069760 tx=0x7f99e402af90 comp rx=0 tx=0).stop 2024-07-26T06:19:25.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 -- 172.21.15.163:0/526548988 shutdown_connections 2024-07-26T06:19:25.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 --2- 172.21.15.163:0/526548988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc06bf40 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 --2- 172.21.15.163:0/526548988 >> v2:172.21.15.164:3300/0 conn(0x7f99fc159380 0x7f99fc0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 -- 172.21.15.163:0/526548988 >> 172.21.15.163:0/526548988 conn(0x7f99fc06d420 msgr2=0x7f99fc06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:25.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 -- 172.21.15.163:0/526548988 shutdown_connections 2024-07-26T06:19:25.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.500+0000 7f9a02e00640 1 -- 172.21.15.163:0/526548988 wait complete. 2024-07-26T06:19:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.502+0000 7f9a02e00640 1 Processor -- start 2024-07-26T06:19:25.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.502+0000 7f9a02e00640 1 -- start start 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f9a02e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99fc06c480 0x7f99fc14a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f9a02e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f9a02e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc159380 0x7f99fc0ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f9a02e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc0cd8f0 con 0x7f99fc0b40a0 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f9a02e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc06f580 con 0x7f99fc06c480 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f9a02e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f99fc0cda70 con 0x7f99fc159380 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f99fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.503+0000 7f99fb400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc0ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55924/0 (socket says 172.21.15.163:55924) 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 -- 172.21.15.163:0/2090797314 learned_addr learned my addr 172.21.15.163:0/2090797314 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 -- 172.21.15.163:0/2090797314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc159380 msgr2=0x7f99fc0ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc159380 0x7f99fc0ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 -- 172.21.15.163:0/2090797314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99fc06c480 msgr2=0x7f99fc14a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99fc06c480 0x7f99fc14a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 -- 172.21.15.163:0/2090797314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc1f4b30 con 0x7f99fc0b40a0 2024-07-26T06:19:25.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99fb400640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc0ad8e0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f99e4000f80 tx=0x7f99e4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.504+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4040020 con 0x7f99fc0b40a0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.505+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99e4004040 con 0x7f99fc0b40a0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.505+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4038040 con 0x7f99fc0b40a0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.505+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc14b8d0 con 0x7f99fc0b40a0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.505+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc151fa0 con 0x7f99fc0b40a0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.506+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99fc06bf40 con 0x7f99fc0b40a0 2024-07-26T06:19:25.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.507+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f99e4043020 con 0x7f99fc0b40a0 2024-07-26T06:19:25.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.508+0000 7f99f3e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f99d8076630 0x7f99d8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.508+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5767+0+0 (secure 0 0 0) 0x7f99e40b7ff0 con 0x7f99fc0b40a0 2024-07-26T06:19:25.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.508+0000 7f99fbe00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f99d8076630 0x7f99d8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.509+0000 7f99fbe00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f99d8076630 0x7f99d8078af0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009670 tx=0x7f99f4006bf0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:25.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.512+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99e404a070 con 0x7f99fc0b40a0 2024-07-26T06:19:25.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.761+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f99fc1526c0 con 0x7f99fc0b40a0 2024-07-26T06:19:25.762 INFO:tasks.ceph.mon.a.smithi163.stderr:2024-07-26T06:19:25.761+0000 7fa697e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-07-26T06:19:25.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.765+0000 7f99f3e00640 1 -- 172.21.15.163:0/2090797314 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) ==== 161+0+0 (secure 0 0 0) 0x7f99fc14b8d0 con 0x7f99fc0b40a0 2024-07-26T06:19:25.766 INFO:teuthology.orchestra.run.smithi163.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-07-26T06:19:25.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f99d8076630 msgr2=0x7f99d8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f99d8076630 0x7f99d8078af0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009670 tx=0x7f99f4006bf0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 msgr2=0x7f99fc0ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc0ad8e0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f99e4000f80 tx=0x7f99e4009900 comp rx=0 tx=0).stop 2024-07-26T06:19:25.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 shutdown_connections 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f99d8076630 0x7f99d8078af0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc159380 0x7f99fc0ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99fc0b40a0 0x7f99fc0ad8e0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 --2- 172.21.15.163:0/2090797314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99fc06c480 0x7f99fc14a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 >> 172.21.15.163:0/2090797314 conn(0x7f99fc06d420 msgr2=0x7f99fc157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 shutdown_connections 2024-07-26T06:19:25.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.768+0000 7f9a02e00640 1 -- 172.21.15.163:0/2090797314 wait complete. 2024-07-26T06:19:25.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-07-26T06:19:25.880 INFO:tasks.ceph.mds.c.smithi163.stdout:starting mds.c at 2024-07-26T06:19:25.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.977+0000 7f2713800640 1 -- 172.21.15.163:0/2222710252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c074590 msgr2=0x7f270c0cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.977+0000 7f2713800640 1 --2- 172.21.15.163:0/2222710252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c074590 0x7f270c0cc630 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f270c0b1eb0 tx=0x7f26fc02a4d0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.977+0000 7f2713800640 1 -- 172.21.15.163:0/2222710252 shutdown_connections 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.978+0000 7f2713800640 1 --2- 172.21.15.163:0/2222710252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f270c1599a0 0x7f270c159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.978+0000 7f2713800640 1 --2- 172.21.15.163:0/2222710252 >> v2:172.21.15.164:3300/0 conn(0x7f270c06d260 0x7f270c06d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.978+0000 7f2713800640 1 --2- 172.21.15.163:0/2222710252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c074590 0x7f270c0cc630 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.978+0000 7f2713800640 1 -- 172.21.15.163:0/2222710252 >> 172.21.15.163:0/2222710252 conn(0x7f270c06bf40 msgr2=0x7f270c06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.978+0000 7f2713800640 1 -- 172.21.15.163:0/2222710252 shutdown_connections 2024-07-26T06:19:25.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.978+0000 7f2713800640 1 -- 172.21.15.163:0/2222710252 wait complete. 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.979+0000 7f2713800640 1 Processor -- start 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.980+0000 7f2713800640 1 -- start start 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.980+0000 7f2713800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 0x7f270c14b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2713800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f270c074590 0x7f270c0ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2713800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f270c1599a0 0x7f270c157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2713800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f270c06f390 con 0x7f270c06d260 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2713800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f270c06f510 con 0x7f270c074590 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2713800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f270c0b2130 con 0x7f270c1599a0 2024-07-26T06:19:25.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 0x7f270c14b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 0x7f270c14b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55958/0 (socket says 172.21.15.163:55958) 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 -- 172.21.15.163:0/3135781400 learned_addr learned my addr 172.21.15.163:0/3135781400 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 -- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f270c1599a0 msgr2=0x7f270c157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f270be00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f270c074590 0x7f270c0ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2711800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f270c1599a0 0x7f270c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f270c1599a0 0x7f270c157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 -- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f270c074590 msgr2=0x7f270c0ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f270c074590 0x7f270c0ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.981+0000 7f2710e00640 1 -- 172.21.15.163:0/3135781400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c157d00 con 0x7f270c06d260 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.982+0000 7f2710e00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 0x7f270c14b8d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009340 tx=0x7f26fc005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.982+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035070 con 0x7f270c06d260 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.982+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26fc041070 con 0x7f270c06d260 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.982+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc003850 con 0x7f270c06d260 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.982+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270c157fe0 con 0x7f270c06d260 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.982+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f270c1585a0 con 0x7f270c06d260 2024-07-26T06:19:25.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.983+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270c069810 con 0x7f270c06d260 2024-07-26T06:19:25.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.985+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f26fc0039f0 con 0x7f270c06d260 2024-07-26T06:19:25.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.986+0000 7f2708c00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f26ec076630 0x7f26ec078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:25.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.986+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5853+0+0 (secure 0 0 0) 0x7f26fc0b8ec0 con 0x7f270c06d260 2024-07-26T06:19:25.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.986+0000 7f270be00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f26ec076630 0x7f26ec078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:25.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.987+0000 7f270be00640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f26ec076630 0x7f26ec078af0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f26f8005d80 tx=0x7f26f8005c90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:25.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:25.990+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26fc082270 con 0x7f270c06d260 2024-07-26T06:19:26.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.239+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f270c06a330 con 0x7f270c06d260 2024-07-26T06:19:26.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.768+0000 7f2708c00640 1 -- 172.21.15.163:0/3135781400 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v73) ==== 104+0+0 (secure 0 0 0) 0x7f270c06a330 con 0x7f270c06d260 2024-07-26T06:19:26.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f26ec076630 msgr2=0x7f26ec078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:26.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f26ec076630 0x7f26ec078af0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f26f8005d80 tx=0x7f26f8005c90 comp rx=0 tx=0).stop 2024-07-26T06:19:26.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 msgr2=0x7f270c14b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:26.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 0x7f270c14b8d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009340 tx=0x7f26fc005770 comp rx=0 tx=0).stop 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 shutdown_connections 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f26ec076630 0x7f26ec078af0 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f26f8005d80 tx=0x7f26f8005c90 comp rx=0 tx=0).stop 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f270c1599a0 0x7f270c157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f270c074590 0x7f270c0ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 --2- 172.21.15.163:0/3135781400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f270c06d260 0x7f270c14b8d0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.772+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 >> 172.21.15.163:0/3135781400 conn(0x7f270c06bf40 msgr2=0x7f270c156a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.773+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 shutdown_connections 2024-07-26T06:19:26.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:26.773+0000 7f2713800640 1 -- 172.21.15.163:0/3135781400 wait complete. 2024-07-26T06:19:26.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.001+0000 7f7117200640 1 -- 172.21.15.163:0/1145451985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 msgr2=0x7f71180b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.001+0000 7f7117200640 1 --2- 172.21.15.163:0/1145451985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180b6a00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7118160e50 tx=0x7f711002a8f0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.001+0000 7f7117200640 1 -- 172.21.15.163:0/1145451985 shutdown_connections 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.001+0000 7f7117200640 1 --2- 172.21.15.163:0/1145451985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180b6a00 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.001+0000 7f7117200640 1 --2- 172.21.15.163:0/1145451985 >> v2:172.21.15.164:3300/0 conn(0x7f71180b1e80 0x7f71180b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.001+0000 7f7117200640 1 -- 172.21.15.163:0/1145451985 >> 172.21.15.163:0/1145451985 conn(0x7f71180cfda0 msgr2=0x7f71180d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.002+0000 7f7117200640 1 -- 172.21.15.163:0/1145451985 shutdown_connections 2024-07-26T06:19:27.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.002+0000 7f7117200640 1 -- 172.21.15.163:0/1145451985 wait complete. 2024-07-26T06:19:27.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.003+0000 7f7117200640 1 Processor -- start 2024-07-26T06:19:27.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.003+0000 7f7117200640 1 -- start start 2024-07-26T06:19:27.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.004+0000 7f7117200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71180b1e80 0x7f7118155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:27.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.004+0000 7f7117200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:27.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.004+0000 7f7117200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f711815d5a0 0x7f71180ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:27.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.004+0000 7f7117200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f71180b4f20 con 0x7f71180b23d0 2024-07-26T06:19:27.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.004+0000 7f7117200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f71180b4da0 con 0x7f711815d5a0 2024-07-26T06:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.004+0000 7f7117200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f71180b50a0 con 0x7f71180b1e80 2024-07-26T06:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55980/0 (socket says 172.21.15.163:55980) 2024-07-26T06:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 -- 172.21.15.163:0/1398513151 learned_addr learned my addr 172.21.15.163:0/1398513151 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:27.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115e00640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71180b1e80 0x7f7118155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:27.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 -- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71180b1e80 msgr2=0x7f7118155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:27.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71180b1e80 0x7f7118155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 -- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f711815d5a0 msgr2=0x7f71180ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f711815d5a0 0x7f71180ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115400640 1 -- 172.21.15.163:0/1398513151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71180c7e60 con 0x7f71180b23d0 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.005+0000 7f7115e00640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71180b1e80 0x7f7118155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.006+0000 7f7115400640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180cd100 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f710000c9a0 tx=0x7f710000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.006+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100007bb0 con 0x7f71180b23d0 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.006+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7100005020 con 0x7f71180b23d0 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.006+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100007e60 con 0x7f71180b23d0 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.006+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71180cd640 con 0x7f71180b23d0 2024-07-26T06:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.006+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71180c83e0 con 0x7f71180b23d0 2024-07-26T06:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.007+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711815a630 con 0x7f71180b23d0 2024-07-26T06:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.008+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f7100005470 con 0x7f71180b23d0 2024-07-26T06:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.009+0000 7f710e000640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70f4076660 0x7f70f4078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.009+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5853+0+0 (secure 0 0 0) 0x7f7100098420 con 0x7f71180b23d0 2024-07-26T06:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.010+0000 7f7115e00640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70f4076660 0x7f70f4078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.010+0000 7f7115e00640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70f4076660 0x7f70f4078b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f710800a810 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.014+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f711815a630 con 0x7f71180b23d0 2024-07-26T06:19:27.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.224+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f71180d1190 con 0x7f71180b23d0 2024-07-26T06:19:27.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:19:27.301 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:19:27.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.772+0000 7f710e000640 1 -- 172.21.15.163:0/1398513151 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) ==== 181+0+0 (secure 0 0 0) 0x7f7100014030 con 0x7f71180b23d0 2024-07-26T06:19:27.774 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_metadata' removed 2024-07-26T06:19:27.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70f4076660 msgr2=0x7f70f4078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:27.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70f4076660 0x7f70f4078b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f710800a810 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-07-26T06:19:27.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 msgr2=0x7f71180cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:27.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180cd100 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f710000c9a0 tx=0x7f710000ce60 comp rx=0 tx=0).stop 2024-07-26T06:19:27.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 shutdown_connections 2024-07-26T06:19:27.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f70f4076660 0x7f70f4078b20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f711815d5a0 0x7f71180ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71180b23d0 0x7f71180cd100 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 --2- 172.21.15.163:0/1398513151 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71180b1e80 0x7f7118155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:27.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.776+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 >> 172.21.15.163:0/1398513151 conn(0x7f71180cfda0 msgr2=0x7f71180b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:27.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.777+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 shutdown_connections 2024-07-26T06:19:27.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:27.777+0000 7f7117200640 1 -- 172.21.15.163:0/1398513151 wait complete. 2024-07-26T06:19:27.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-07-26T06:19:28.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 -- 172.21.15.163:0/3768122632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c159370 msgr2=0x7fe31c159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:28.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 --2- 172.21.15.163:0/3768122632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c159370 0x7fe31c159750 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe30c005fd0 tx=0x7fe30c02a540 comp rx=0 tx=0).stop 2024-07-26T06:19:28.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 -- 172.21.15.163:0/3768122632 shutdown_connections 2024-07-26T06:19:28.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 --2- 172.21.15.163:0/3768122632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31c0cbad0 0x7fe31c0cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 --2- 172.21.15.163:0/3768122632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c159370 0x7fe31c159750 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 -- 172.21.15.163:0/3768122632 >> 172.21.15.163:0/3768122632 conn(0x7fe31c0b0e30 msgr2=0x7fe31c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:28.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 -- 172.21.15.163:0/3768122632 shutdown_connections 2024-07-26T06:19:28.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.009+0000 7fe31be00640 1 -- 172.21.15.163:0/3768122632 wait complete. 2024-07-26T06:19:28.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.011+0000 7fe31be00640 1 Processor -- start 2024-07-26T06:19:28.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.011+0000 7fe31be00640 1 -- start start 2024-07-26T06:19:28.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31be00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 0x7fe31c137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:28.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31be00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31c159370 0x7fe31c1508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:28.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31be00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31c15bfe0 0x7fe31c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:28.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31be00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c0cda70 con 0x7fe31c0b4420 2024-07-26T06:19:28.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31be00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c06f500 con 0x7fe31c159370 2024-07-26T06:19:28.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31be00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe31c0cdbf0 con 0x7fe31c15bfe0 2024-07-26T06:19:28.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 0x7fe31c137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:28.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31aa00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 0x7fe31c137880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55988/0 (socket says 172.21.15.163:55988) 2024-07-26T06:19:28.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.012+0000 7fe31aa00640 1 -- 172.21.15.163:0/1861231388 learned_addr learned my addr 172.21.15.163:0/1861231388 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:28.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31aa00640 1 -- 172.21.15.163:0/1861231388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31c15bfe0 msgr2=0x7fe31c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:28.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31aa00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31c15bfe0 0x7fe31c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31aa00640 1 -- 172.21.15.163:0/1861231388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31c159370 msgr2=0x7fe31c1508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:28.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31aa00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31c159370 0x7fe31c1508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31aa00640 1 -- 172.21.15.163:0/1861231388 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c14c3e0 con 0x7fe31c0b4420 2024-07-26T06:19:28.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31aa00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 0x7fe31c137880 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe30c034040 tx=0x7fe30c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:28.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c03e030 con 0x7fe31c0b4420 2024-07-26T06:19:28.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe30c003d90 con 0x7fe31c0b4420 2024-07-26T06:19:28.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c035040 con 0x7fe31c0b4420 2024-07-26T06:19:28.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c151310 con 0x7fe31c0b4420 2024-07-26T06:19:28.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.013+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c1518d0 con 0x7fe31c0b4420 2024-07-26T06:19:28.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.015+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c0cd530 con 0x7fe31c0b4420 2024-07-26T06:19:28.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.016+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fe30c041030 con 0x7fe31c0b4420 2024-07-26T06:19:28.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.017+0000 7fe312a00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe2fc076630 0x7fe2fc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:28.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.017+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5415+0+0 (secure 0 0 0) 0x7fe30c0b8330 con 0x7fe31c0b4420 2024-07-26T06:19:28.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.017+0000 7fe31a000640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe2fc076630 0x7fe2fc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:28.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.018+0000 7fe31a000640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe2fc076630 0x7fe2fc078af0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe308005fd0 tx=0x7fe308005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:28.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.021+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe30c081890 con 0x7fe31c0b4420 2024-07-26T06:19:28.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.225+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe31c152230 con 0x7fe31c0b4420 2024-07-26T06:19:28.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.785+0000 7fe312a00640 1 -- 172.21.15.163:0/1861231388 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) ==== 169+0+0 (secure 0 0 0) 0x7fe30c081190 con 0x7fe31c0b4420 2024-07-26T06:19:28.787 INFO:teuthology.orchestra.run.smithi163.stderr:pool 'cephfs_data' removed 2024-07-26T06:19:28.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.789+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe2fc076630 msgr2=0x7fe2fc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.789+0000 7fe31be00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe2fc076630 0x7fe2fc078af0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe308005fd0 tx=0x7fe308005850 comp rx=0 tx=0).stop 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.789+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 msgr2=0x7fe31c137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.789+0000 7fe31be00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 0x7fe31c137880 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe30c034040 tx=0x7fe30c0038a0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 shutdown_connections 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe2fc076630 0x7fe2fc078af0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31c15bfe0 0x7fe31c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31c159370 0x7fe31c1508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 --2- 172.21.15.163:0/1861231388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe31c0b4420 0x7fe31c137880 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:28.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 >> 172.21.15.163:0/1861231388 conn(0x7fe31c0b0e30 msgr2=0x7fe31c152750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.790+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 shutdown_connections 2024-07-26T06:19:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:28.791+0000 7fe31be00640 1 -- 172.21.15.163:0/1861231388 wait complete. 2024-07-26T06:19:28.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-07-26T06:19:29.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 -- 172.21.15.163:0/1821256618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 msgr2=0x7f1a9815ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/1821256618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 0x7f1a9815ed20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009830 tx=0x7f1a8002a480 comp rx=0 tx=0).stop 2024-07-26T06:19:29.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 -- 172.21.15.163:0/1821256618 shutdown_connections 2024-07-26T06:19:29.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/1821256618 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a9806bf40 0x7f1a9806c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/1821256618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 0x7f1a9815ed20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 -- 172.21.15.163:0/1821256618 >> 172.21.15.163:0/1821256618 conn(0x7f1a9806d420 msgr2=0x7f1a9806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:29.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 -- 172.21.15.163:0/1821256618 shutdown_connections 2024-07-26T06:19:29.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.015+0000 7f1a9ce00640 1 -- 172.21.15.163:0/1821256618 wait complete. 2024-07-26T06:19:29.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.017+0000 7f1a9ce00640 1 Processor -- start 2024-07-26T06:19:29.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.017+0000 7f1a9ce00640 1 -- start start 2024-07-26T06:19:29.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a9ce00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a9806bf40 0x7f1a9814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a9ce00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 0x7f1a9806d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a9ce00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 0x7f1a9806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a9ce00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a980736b0 con 0x7f1a980cb7e0 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a9ce00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1a98073530 con 0x7f1a98071cf0 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a9ce00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1a98073830 con 0x7f1a9806bf40 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 0x7f1a9806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 0x7f1a9806dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55992/0 (socket says 172.21.15.163:55992) 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 -- 172.21.15.163:0/2989702426 learned_addr learned my addr 172.21.15.163:0/2989702426 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a95400640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 0x7f1a9806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a95e00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a9806bf40 0x7f1a9814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 -- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a9806bf40 msgr2=0x7f1a9814df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a9806bf40 0x7f1a9814df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 -- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 msgr2=0x7f1a9806d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.019+0000 7f1a96800640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 0x7f1a9806d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a96800640 1 -- 172.21.15.163:0/2989702426 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a981f3bc0 con 0x7f1a980cb7e0 2024-07-26T06:19:29.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a96800640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 0x7f1a9806dd40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000efc0 tx=0x7f1a9000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:29.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90003970 con 0x7f1a980cb7e0 2024-07-26T06:19:29.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a90003c20 con 0x7f1a980cb7e0 2024-07-26T06:19:29.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a980af0e0 con 0x7f1a980cb7e0 2024-07-26T06:19:29.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90004280 con 0x7f1a980cb7e0 2024-07-26T06:19:29.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.020+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a980af6a0 con 0x7f1a980cb7e0 2024-07-26T06:19:29.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.021+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a98073110 con 0x7f1a980cb7e0 2024-07-26T06:19:29.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.023+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f1a9001d020 con 0x7f1a980cb7e0 2024-07-26T06:19:29.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.024+0000 7f1a8e000640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a74076630 0x7f1a74078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.024+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f1a90097fd0 con 0x7f1a980cb7e0 2024-07-26T06:19:29.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.024+0000 7f1a95e00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a74076630 0x7f1a74078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.025+0000 7f1a95e00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a74076630 0x7f1a74078af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1a98155dc0 tx=0x7f1a88009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:29.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.029+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a98073110 con 0x7f1a980cb7e0 2024-07-26T06:19:29.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.235+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f1a98073d00 con 0x7f1a980cb7e0 2024-07-26T06:19:29.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.241+0000 7f1a8e000640 1 -- 172.21.15.163:0/2989702426 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v39) ==== 71+0+0 (secure 0 0 0) 0x7f1a900617a0 con 0x7f1a980cb7e0 2024-07-26T06:19:29.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.244+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a74076630 msgr2=0x7f1a74078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.244+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a74076630 0x7f1a74078af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1a98155dc0 tx=0x7f1a88009040 comp rx=0 tx=0).stop 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.244+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 msgr2=0x7f1a9806dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.244+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 0x7f1a9806dd40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000efc0 tx=0x7f1a9000c800 comp rx=0 tx=0).stop 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 shutdown_connections 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f1a74076630 0x7f1a74078af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a980cb7e0 0x7f1a9806dd40 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a98071cf0 0x7f1a9806d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 --2- 172.21.15.163:0/2989702426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a9806bf40 0x7f1a9814df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 >> 172.21.15.163:0/2989702426 conn(0x7f1a9806d420 msgr2=0x7f1a980cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 shutdown_connections 2024-07-26T06:19:29.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.245+0000 7f1a9ce00640 1 -- 172.21.15.163:0/2989702426 wait complete. 2024-07-26T06:19:29.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 -- 172.21.15.163:0/3462016004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21b40052e0 msgr2=0x7f21b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 --2- 172.21.15.163:0/3462016004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21b40052e0 0x7f21b400b9c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f21c8001c50 tx=0x7f21b4005830 comp rx=0 tx=0).stop 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 -- 172.21.15.163:0/3462016004 shutdown_connections 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 --2- 172.21.15.163:0/3462016004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21b40052e0 0x7f21b400b9c0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 -- 172.21.15.163:0/3462016004 >> 172.21.15.163:0/3462016004 conn(0x7f21d00d36b0 msgr2=0x7f21d00d3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 -- 172.21.15.163:0/3462016004 shutdown_connections 2024-07-26T06:19:29.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.470+0000 7f21d7200640 1 -- 172.21.15.163:0/3462016004 wait complete. 2024-07-26T06:19:29.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.471+0000 7f21d7200640 1 Processor -- start 2024-07-26T06:19:29.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.472+0000 7f21d7200640 1 -- start start 2024-07-26T06:19:29.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.472+0000 7f21d7200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21b40052e0 0x7f21d00b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.472+0000 7f21d7200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21d0071050 0x7f21d00714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d7200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 0x7f21d01fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d7200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f21d00b5980 con 0x7f21d01fb6b0 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d7200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f21d00b5800 con 0x7f21d0071050 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d7200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f21d00b5b00 con 0x7f21b40052e0 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 0x7f21d01fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 0x7f21d01fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:55998/0 (socket says 172.21.15.163:55998) 2024-07-26T06:19:29.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 -- 172.21.15.163:0/3940896429 learned_addr learned my addr 172.21.15.163:0/3940896429 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:29.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21cfe00640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21b40052e0 0x7f21d00b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 -- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21b40052e0 msgr2=0x7f21d00b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21b40052e0 0x7f21d00b7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 -- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21d0071050 msgr2=0x7f21d00714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21c7400640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21d0071050 0x7f21d00714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21d0071050 0x7f21d00714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.473+0000 7f21d5200640 1 -- 172.21.15.163:0/3940896429 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d00cb8c0 con 0x7f21d01fb6b0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.474+0000 7f21d5200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 0x7f21d01fbb20 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000f80 tx=0x7f21c80066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.474+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c80068a0 con 0x7f21d01fb6b0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.474+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21c803e760 con 0x7f21d01fb6b0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.474+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d00cbba0 con 0x7f21d01fb6b0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.474+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8006c80 con 0x7f21d01fb6b0 2024-07-26T06:19:29.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.474+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d00cc160 con 0x7f21d01fb6b0 2024-07-26T06:19:29.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.475+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d00d1a20 con 0x7f21d01fb6b0 2024-07-26T06:19:29.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.476+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f21c8033440 con 0x7f21d01fb6b0 2024-07-26T06:19:29.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.477+0000 7f21ccc00640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21ac076660 0x7f21ac078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:29.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.478+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f21c80b8070 con 0x7f21d01fb6b0 2024-07-26T06:19:29.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.478+0000 7f21cfe00640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21ac076660 0x7f21ac078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:29.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.478+0000 7f21cfe00640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21ac076660 0x7f21ac078b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21c00057c0 tx=0x7f21c0005cf0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:29.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.482+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f21c80816b0 con 0x7f21d01fb6b0 2024-07-26T06:19:29.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.779+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7f21d015cdd0 con 0x7f21d01fb6b0 2024-07-26T06:19:29.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.788+0000 7f21ccc00640 1 -- 172.21.15.163:0/3940896429 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v316) ==== 149+0+0 (secure 0 0 0) 0x7f21c8080fb0 con 0x7f21d01fb6b0 2024-07-26T06:19:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21ac076660 msgr2=0x7f21ac078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21ac076660 0x7f21ac078b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21c00057c0 tx=0x7f21c0005cf0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 msgr2=0x7f21d01fbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 0x7f21d01fbb20 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000f80 tx=0x7f21c80066b0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 shutdown_connections 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f21ac076660 0x7f21ac078b20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f21d01fb6b0 0x7f21d01fbb20 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21d0071050 0x7f21d00714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 --2- 172.21.15.163:0/3940896429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21b40052e0 0x7f21d00b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 >> 172.21.15.163:0/3940896429 conn(0x7f21d00d36b0 msgr2=0x7f21d00b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 shutdown_connections 2024-07-26T06:19:29.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:29.791+0000 7f21d7200640 1 -- 172.21.15.163:0/3940896429 wait complete. 2024-07-26T06:19:29.817 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ERROR 2024-07-26T06:19:29.817 INFO:tasks.cephfs_test_runner: 2024-07-26T06:19:29.817 INFO:tasks.cephfs_test_runner:====================================================================== 2024-07-26T06:19:29.817 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:19:29.817 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... 2024-07-26T06:19:29.817 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 8629, in test_subvolume_no_upgrade_v1_sanity 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: self._wait_for_clone_to_complete(clone1) 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 93, in _wait_for_clone_to_complete 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: self.__check_clone_state("complete", clone, clone_group, timo) 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 60, in __check_clone_state 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: result = json.loads(self._fs_cmd(*args)) 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 41, in _fs_cmd 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/ceph_manager.py", line 1652, in run_cluster_cmd 2024-07-26T06:19:29.818 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: r.wait() 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 95: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone33550' 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner:Ran 6 tests in 325.418s 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner: 2024-07-26T06:19:29.819 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner:====================================================================== 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 8629, in test_subvolume_no_upgrade_v1_sanity 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: self._wait_for_clone_to_complete(clone1) 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 93, in _wait_for_clone_to_complete 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: self.__check_clone_state("complete", clone, clone_group, timo) 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 60, in __check_clone_state 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: result = json.loads(self._fs_cmd(*args)) 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs/test_volumes.py", line 41, in _fs_cmd 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-07-26T06:19:29.820 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/ceph_manager.py", line 1652, in run_cluster_cmd 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: r.wait() 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 95: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone33550' 2024-07-26T06:19:29.821 INFO:tasks.cephfs_test_runner: 2024-07-26T06:19:29.822 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:19:35.588 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d1bfbf22bbe944c8a893c46e866aba68 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4aae82f07003420ccc152fe0b01f3d5a44885708/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-07-26T06:19:35.590 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-07-26T06:19:35.600 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-07-26T06:19:35.613 INFO:tasks.kclient:Unmounting kernel clients... 2024-07-26T06:19:35.613 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:35.613 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:35.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:19:35.674 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:19:35.675 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:35.725 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052680,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-07-26T06:19:35.726 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:19:35.726 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:35.785 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:19:35.785 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:19:35.785 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:19:35.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-07-26T06:19:35.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:35.836 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:19:35.894 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-07-26T06:19:35.894 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-07-26T06:19:35.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:35.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:19:35.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:35.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-07-26T06:19:36.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:36.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link delete ceph-brx 2024-07-26T06:19:36.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:36.712 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-07-26T06:19:36.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:36.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link delete ceph-brx 2024-07-26T06:19:36.887 INFO:teuthology.orchestra.run.smithi163.stderr:Cannot find device "ceph-brx" 2024-07-26T06:19:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:19:36.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:36.891 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:19:36.947 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-07-26T06:19:36.948 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-07-26T06:19:36.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:36.949 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-07-26T06:19:37.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:37.039 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-07-26T06:19:37.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:37.414 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link delete ceph-brx 2024-07-26T06:19:37.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:37.723 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-07-26T06:19:37.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:19:37.766 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link delete ceph-brx 2024-07-26T06:19:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:Cannot find device "ceph-brx" 2024-07-26T06:19:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:19:37.904 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-07-26T06:19:37.914 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-07-26T06:19:37.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:19:38.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.128+0000 7fb591c00640 1 -- 172.21.15.163:0/3477793604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5700039e0 msgr2=0x7fb570003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.128+0000 7fb591c00640 1 --2- 172.21.15.163:0/3477793604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb5700039e0 0x7fb570003de0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c310 tx=0x7fb570005830 comp rx=0 tx=0).stop 2024-07-26T06:19:38.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.128+0000 7fb591200640 1 -- 172.21.15.163:0/3477793604 reap_dead start 2024-07-26T06:19:38.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.128+0000 7fb591c00640 1 -- 172.21.15.163:0/3477793604 shutdown_connections 2024-07-26T06:19:38.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.129+0000 7fb591c00640 1 -- 172.21.15.163:0/3477793604 >> 172.21.15.163:0/3477793604 conn(0x7fb58c0b0e30 msgr2=0x7fb58c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:38.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.129+0000 7fb591c00640 1 -- 172.21.15.163:0/3477793604 shutdown_connections 2024-07-26T06:19:38.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.129+0000 7fb591c00640 1 -- 172.21.15.163:0/3477793604 wait complete. 2024-07-26T06:19:38.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.130+0000 7fb591c00640 1 Processor -- start 2024-07-26T06:19:38.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.130+0000 7fb591c00640 1 -- start start 2024-07-26T06:19:38.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.131+0000 7fb591c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5700039e0 0x7fb58c14c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.131+0000 7fb591c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 0x7fb58c15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.131+0000 7fb591c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb58c1537e0 0x7fb58c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.131+0000 7fb591c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c0b3e20 con 0x7fb58c159ea0 2024-07-26T06:19:38.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.131+0000 7fb591c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c071780 con 0x7fb58c1537e0 2024-07-26T06:19:38.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.131+0000 7fb591c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb58c0b3fa0 con 0x7fb5700039e0 2024-07-26T06:19:38.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 0x7fb58c15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:38.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 0x7fb58c15a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34146/0 (socket says 172.21.15.163:34146) 2024-07-26T06:19:38.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 -- 172.21.15.163:0/2982873142 learned_addr learned my addr 172.21.15.163:0/2982873142 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 -- 172.21.15.163:0/2982873142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5700039e0 msgr2=0x7fb58c14c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5700039e0 0x7fb58c14c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 -- 172.21.15.163:0/2982873142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb58c1537e0 msgr2=0x7fb58c153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb58c1537e0 0x7fb58c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 -- 172.21.15.163:0/2982873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c158440 con 0x7fb58c159ea0 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.132+0000 7fb583400640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 0x7fb58c15a320 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb57400e6e0 tx=0x7fb57400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:38.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.133+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57400ca50 con 0x7fb58c159ea0 2024-07-26T06:19:38.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.133+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb57400cce0 con 0x7fb58c159ea0 2024-07-26T06:19:38.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.133+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5740116d0 con 0x7fb58c159ea0 2024-07-26T06:19:38.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.133+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb58c158720 con 0x7fb58c159ea0 2024-07-26T06:19:38.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.133+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb58c158ce0 con 0x7fb58c159ea0 2024-07-26T06:19:38.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.134+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58c0b4500 con 0x7fb58c159ea0 2024-07-26T06:19:38.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.135+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fb574011870 con 0x7fb58c159ea0 2024-07-26T06:19:38.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.136+0000 7fb588c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb568076660 0x7fb568078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.136+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fb574098450 con 0x7fb58c159ea0 2024-07-26T06:19:38.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.136+0000 7fb58be00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb568076660 0x7fb568078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:38.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.137+0000 7fb58be00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb568076660 0x7fb568078b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb57c033000 tx=0x7fb57c001a30 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:38.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.140+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb574016030 con 0x7fb58c159ea0 2024-07-26T06:19:38.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.346+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb58c0b34f0 con 0x7fb568076660 2024-07-26T06:19:38.349 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:38.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.347+0000 7fb588c00640 1 -- 172.21.15.163:0/2982873142 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13412 (secure 0 0 0) 0x7fb58c0b34f0 con 0x7fb568076660 2024-07-26T06:19:38.349 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:19:38.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb568076660 msgr2=0x7fb568078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb568076660 0x7fb568078b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb57c033000 tx=0x7fb57c001a30 comp rx=0 tx=0).stop 2024-07-26T06:19:38.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 msgr2=0x7fb58c15a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 0x7fb58c15a320 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb57400e6e0 tx=0x7fb57400eba0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 shutdown_connections 2024-07-26T06:19:38.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb568076660 0x7fb568078b20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb58c1537e0 0x7fb58c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb58c159ea0 0x7fb58c15a320 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 --2- 172.21.15.163:0/2982873142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5700039e0 0x7fb58c14c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 >> 172.21.15.163:0/2982873142 conn(0x7fb58c0b0e30 msgr2=0x7fb58c151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:38.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 shutdown_connections 2024-07-26T06:19:38.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.350+0000 7fb591c00640 1 -- 172.21.15.163:0/2982873142 wait complete. 2024-07-26T06:19:38.371 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-07-26T06:19:36.485645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260732,"kb_used_data":11272,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713988,"statfs":{"total":773094113280,"available":772827123712,"internally_reserved":0,"allocated":11542528,"data_stored":5697104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16777478,"num_objects":-18,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-18,"num_whiteouts":0,"num_read":-45,"num_read_kb":-8222,"num_write":-28,"num_write_kb":-16391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.709913"},"pg_stats":[{"pgid":"1.0","version":"70'79","reported_seq":291,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-07-26T06:19:30.862750+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:19:30.862750+0000","last_peered":"2024-07-26T06:19:30.862750+0000","last_clean":"2024-07-26T06:19:30.862750+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:19:30.862750+0000","last_undegraded":"2024-07-26T06:19:30.862750+0000","last_fullsized":"2024-07-26T06:19:30.862750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:19:38.372 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:19:38.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.582+0000 7f8006c00640 1 -- 172.21.15.163:0/2123485554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cc3c0 msgr2=0x7f800015b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.582+0000 7f8006c00640 1 --2- 172.21.15.163:0/2123485554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cc3c0 0x7f800015b240 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f8000070bc0 tx=0x7f7ff802aac0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.582+0000 7f8006c00640 1 -- 172.21.15.163:0/2123485554 shutdown_connections 2024-07-26T06:19:38.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.582+0000 7f8006c00640 1 --2- 172.21.15.163:0/2123485554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cc3c0 0x7f800015b240 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.582+0000 7f8006c00640 1 --2- 172.21.15.163:0/2123485554 >> v2:172.21.15.164:3300/0 conn(0x7f80001507d0 0x7f80000ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.582+0000 7f8006c00640 1 -- 172.21.15.163:0/2123485554 >> 172.21.15.163:0/2123485554 conn(0x7f800006d420 msgr2=0x7f800006d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:38.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.583+0000 7f8006c00640 1 -- 172.21.15.163:0/2123485554 shutdown_connections 2024-07-26T06:19:38.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.583+0000 7f8006c00640 1 -- 172.21.15.163:0/2123485554 wait complete. 2024-07-26T06:19:38.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.584+0000 7f8006c00640 1 Processor -- start 2024-07-26T06:19:38.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.584+0000 7f8006c00640 1 -- start start 2024-07-26T06:19:38.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.585+0000 7f8006c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80000cc3c0 0x7f800014ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.585+0000 7f8006c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 0x7f800014df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.585+0000 7f8006c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80001507d0 0x7f80000b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.585+0000 7f8006c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f800006c410 con 0x7f80000cd480 2024-07-26T06:19:38.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.585+0000 7f8006c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f800006c290 con 0x7f80001507d0 2024-07-26T06:19:38.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.585+0000 7f8006c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f800006c590 con 0x7f80000cc3c0 2024-07-26T06:19:38.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 0x7f800014df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:38.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 0x7f800014df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34154/0 (socket says 172.21.15.163:34154) 2024-07-26T06:19:38.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 -- 172.21.15.163:0/2064928605 learned_addr learned my addr 172.21.15.163:0/2064928605 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:38.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f8004c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80001507d0 0x7f80000b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:38.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fffe00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80000cc3c0 0x7f800014ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:38.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 -- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80000cc3c0 msgr2=0x7f800014ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80000cc3c0 0x7f800014ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 -- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80001507d0 msgr2=0x7f80000b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80001507d0 0x7f80000b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f7fff400640 1 -- 172.21.15.163:0/2064928605 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80001f4340 con 0x7f80000cd480 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.586+0000 7f8004c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80001507d0 0x7f80000b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.587+0000 7f7fff400640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 0x7f800014df30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c450 tx=0x7f7fec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:38.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.587+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0052d0 con 0x7f80000cd480 2024-07-26T06:19:38.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.587+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fec005470 con 0x7f80000cd480 2024-07-26T06:19:38.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.587+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80001f4620 con 0x7f80000cd480 2024-07-26T06:19:38.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.587+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec012870 con 0x7f80000cd480 2024-07-26T06:19:38.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.587+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80001f4be0 con 0x7f80000cd480 2024-07-26T06:19:38.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.588+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80000740f0 con 0x7f80000cd480 2024-07-26T06:19:38.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.589+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f7fec0035f0 con 0x7f80000cd480 2024-07-26T06:19:38.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.590+0000 7f7ff7e00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7fe0076630 0x7f7fe0078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:38.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.591+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f7fec0975d0 con 0x7f80000cd480 2024-07-26T06:19:38.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.591+0000 7f7fffe00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7fe0076630 0x7f7fe0078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:38.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.591+0000 7f7fffe00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7fe0076630 0x7f7fe0078af0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0005fd0 tx=0x7f7ff0005da0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:38.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.595+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f80000740f0 con 0x7f80000cd480 2024-07-26T06:19:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.798+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f800014b3f0 con 0x7f7fe0076630 2024-07-26T06:19:38.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.799+0000 7f7ff7e00640 1 -- 172.21.15.163:0/2064928605 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13413 (secure 0 0 0) 0x7f800014b3f0 con 0x7f7fe0076630 2024-07-26T06:19:38.800 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:38.801 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:19:38.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7fe0076630 msgr2=0x7f7fe0078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7fe0076630 0x7f7fe0078af0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0005fd0 tx=0x7f7ff0005da0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 msgr2=0x7f800014df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 0x7f800014df30 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c450 tx=0x7f7fec00c910 comp rx=0 tx=0).stop 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 shutdown_connections 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7fe0076630 0x7f7fe0078af0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80001507d0 0x7f80000b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80000cd480 0x7f800014df30 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 --2- 172.21.15.163:0/2064928605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80000cc3c0 0x7f800014ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:38.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 >> 172.21.15.163:0/2064928605 conn(0x7f800006d420 msgr2=0x7f80000b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:38.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.802+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 shutdown_connections 2024-07-26T06:19:38.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:38.803+0000 7f8006c00640 1 -- 172.21.15.163:0/2064928605 wait complete. 2024-07-26T06:19:38.823 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-07-26T06:19:38.486214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260732,"kb_used_data":11272,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713988,"statfs":{"total":773094113280,"available":772827123712,"internally_reserved":0,"allocated":11542528,"data_stored":5697104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16777478,"num_objects":-18,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-18,"num_whiteouts":0,"num_read":-45,"num_read_kb":-8222,"num_write":-28,"num_write_kb":-16391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002958"},"pg_stats":[{"pgid":"1.0","version":"70'79","reported_seq":291,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-07-26T06:19:30.862750+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:19:30.862750+0000","last_peered":"2024-07-26T06:19:30.862750+0000","last_clean":"2024-07-26T06:19:30.862750+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:19:30.862750+0000","last_undegraded":"2024-07-26T06:19:30.862750+0000","last_fullsized":"2024-07-26T06:19:30.862750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:19:38.824 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-07-26T06:19:38.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:19:39.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 -- 172.21.15.163:0/1332120534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 msgr2=0x7f26140ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 --2- 172.21.15.163:0/1332120534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f26140ce2f0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f261406f160 tx=0x7f260402a4a0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 -- 172.21.15.163:0/1332120534 shutdown_connections 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 --2- 172.21.15.163:0/1332120534 >> v2:172.21.15.164:3300/0 conn(0x7f26140b1e80 0x7f26140b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 --2- 172.21.15.163:0/1332120534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f26140ce2f0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 -- 172.21.15.163:0/1332120534 >> 172.21.15.163:0/1332120534 conn(0x7f26140cfda0 msgr2=0x7f26140d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 -- 172.21.15.163:0/1332120534 shutdown_connections 2024-07-26T06:19:39.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.030+0000 7f2613200640 1 -- 172.21.15.163:0/1332120534 wait complete. 2024-07-26T06:19:39.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.032+0000 7f2613200640 1 Processor -- start 2024-07-26T06:19:39.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.032+0000 7f2613200640 1 -- start start 2024-07-26T06:19:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.033+0000 7f2613200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26140b1e80 0x7f26140c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.033+0000 7f2613200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26140ce830 0x7f26140ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.033+0000 7f2613200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f2614149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.033+0000 7f2613200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26140b5130 con 0x7f261415d5a0 2024-07-26T06:19:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.033+0000 7f2613200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f26140b4fb0 con 0x7f26140b1e80 2024-07-26T06:19:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.033+0000 7f2613200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f26140b52b0 con 0x7f26140ce830 2024-07-26T06:19:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f2614149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f2614149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34180/0 (socket says 172.21.15.163:34180) 2024-07-26T06:19:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 -- 172.21.15.163:0/1684655977 learned_addr learned my addr 172.21.15.163:0/1684655977 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:39.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2611e00640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26140b1e80 0x7f26140c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2611400640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26140ce830 0x7f26140ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 -- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26140ce830 msgr2=0x7f26140ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26140ce830 0x7f26140ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 -- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26140b1e80 msgr2=0x7f26140c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26140b1e80 0x7f26140c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 -- 172.21.15.163:0/1684655977 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26140c92c0 con 0x7f261415d5a0 2024-07-26T06:19:39.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2611400640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26140ce830 0x7f26140ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.034+0000 7f2612800640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f2614149f10 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f260c00cb00 tx=0x7f260c00a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.035+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c004b20 con 0x7f261415d5a0 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.035+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f260c004db0 con 0x7f261415d5a0 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.035+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26140c95a0 con 0x7f261415d5a0 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.035+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c005280 con 0x7f261415d5a0 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.035+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26140c9b60 con 0x7f261415d5a0 2024-07-26T06:19:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.036+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26140c6bf0 con 0x7f261415d5a0 2024-07-26T06:19:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.037+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f260c005420 con 0x7f261415d5a0 2024-07-26T06:19:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.038+0000 7f260a000640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f25f4076630 0x7f25f4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.038+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f260c0982d0 con 0x7f261415d5a0 2024-07-26T06:19:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.038+0000 7f2611e00640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f25f4076630 0x7f25f4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.039+0000 7f2611e00640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f25f4076630 0x7f25f4078af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2604034000 tx=0x7f2604004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:39.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.042+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f260c014030 con 0x7f261415d5a0 2024-07-26T06:19:39.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.242+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26140b5820 con 0x7f25f4076630 2024-07-26T06:19:39.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.244+0000 7f260a000640 1 -- 172.21.15.163:0/1684655977 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13413 (secure 0 0 0) 0x7f26140b5820 con 0x7f25f4076630 2024-07-26T06:19:39.245 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:39.245 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:19:39.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.246+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f25f4076630 msgr2=0x7f25f4078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.246+0000 7f2613200640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f25f4076630 0x7f25f4078af0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2604034000 tx=0x7f2604004730 comp rx=0 tx=0).stop 2024-07-26T06:19:39.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.246+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 msgr2=0x7f2614149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.246+0000 7f2613200640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f2614149f10 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f260c00cb00 tx=0x7f260c00a250 comp rx=0 tx=0).stop 2024-07-26T06:19:39.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 shutdown_connections 2024-07-26T06:19:39.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f25f4076630 0x7f25f4078af0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f261415d5a0 0x7f2614149f10 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26140ce830 0x7f26140ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 --2- 172.21.15.163:0/1684655977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26140b1e80 0x7f26140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 >> 172.21.15.163:0/1684655977 conn(0x7f26140cfda0 msgr2=0x7f26140d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:39.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 shutdown_connections 2024-07-26T06:19:39.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.247+0000 7f2613200640 1 -- 172.21.15.163:0/1684655977 wait complete. 2024-07-26T06:19:39.266 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-07-26T06:19:38.486214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260732,"kb_used_data":11272,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713988,"statfs":{"total":773094113280,"available":772827123712,"internally_reserved":0,"allocated":11542528,"data_stored":5697104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16777478,"num_objects":-18,"num_object_clones":0,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-18,"num_whiteouts":0,"num_read":-45,"num_read_kb":-8222,"num_write":-28,"num_write_kb":-16391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002958"},"pg_stats":[{"pgid":"1.0","version":"70'79","reported_seq":291,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-07-26T06:19:30.862750+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:19:30.862750+0000","last_peered":"2024-07-26T06:19:30.862750+0000","last_clean":"2024-07-26T06:19:30.862750+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:19:30.862750+0000","last_undegraded":"2024-07-26T06:19:30.862750+0000","last_fullsized":"2024-07-26T06:19:30.862750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:19:39.267 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T06:19:39.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105e00640 1 -- 172.21.15.163:0/4133777572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 msgr2=0x7f80e4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105e00640 1 --2- 172.21.15.163:0/4133777572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f80e4005210 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009ed0 tx=0x7f80e4005830 comp rx=0 tx=0).stop 2024-07-26T06:19:39.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105400640 1 -- 172.21.15.163:0/4133777572 reap_dead start 2024-07-26T06:19:39.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105e00640 1 -- 172.21.15.163:0/4133777572 shutdown_connections 2024-07-26T06:19:39.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105e00640 1 -- 172.21.15.163:0/4133777572 >> 172.21.15.163:0/4133777572 conn(0x7f81000b0e30 msgr2=0x7f81000b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:39.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105e00640 1 -- 172.21.15.163:0/4133777572 shutdown_connections 2024-07-26T06:19:39.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.464+0000 7f8105e00640 1 -- 172.21.15.163:0/4133777572 wait complete. 2024-07-26T06:19:39.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.466+0000 7f8105e00640 1 Processor -- start 2024-07-26T06:19:39.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.466+0000 7f8105e00640 1 -- start start 2024-07-26T06:19:39.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8105e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f81001f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8105e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81001f5460 0x7f810014bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8105e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000cc590 0x7f81000ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8105e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8100164310 con 0x7f80e4004d90 2024-07-26T06:19:39.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8105e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8100164190 con 0x7f81001f5460 2024-07-26T06:19:39.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8105e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8100164490 con 0x7f81000cc590 2024-07-26T06:19:39.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8104a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f81001f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8104a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f81001f7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34200/0 (socket says 172.21.15.163:34200) 2024-07-26T06:19:39.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.467+0000 7f8104a00640 1 -- 172.21.15.163:0/1947437390 learned_addr learned my addr 172.21.15.163:0/1947437390 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:39.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f80ffe00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81001f5460 0x7f810014bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f8104a00640 1 -- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000cc590 msgr2=0x7f81000ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f8104a00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000cc590 0x7f81000ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f8104a00640 1 -- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81001f5460 msgr2=0x7f810014bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f8104a00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81001f5460 0x7f810014bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f8104a00640 1 -- 172.21.15.163:0/1947437390 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81000ccd70 con 0x7f80e4004d90 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f80ffe00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81001f5460 0x7f810014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:39.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.468+0000 7f8104a00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f81001f7220 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e540 tx=0x7f80f000ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:39.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.469+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0015070 con 0x7f80e4004d90 2024-07-26T06:19:39.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.469+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80f000c9e0 con 0x7f80e4004d90 2024-07-26T06:19:39.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.469+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f000ce30 con 0x7f80e4004d90 2024-07-26T06:19:39.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.469+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81000cca90 con 0x7f80e4004d90 2024-07-26T06:19:39.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.469+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81000aec30 con 0x7f80e4004d90 2024-07-26T06:19:39.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.470+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8100071530 con 0x7f80e4004d90 2024-07-26T06:19:39.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.471+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f80f0018070 con 0x7f80e4004d90 2024-07-26T06:19:39.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.472+0000 7f80fcc00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80e0076630 0x7f80e0078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:39.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.472+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f80f0097f60 con 0x7f80e4004d90 2024-07-26T06:19:39.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.472+0000 7f80ffe00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80e0076630 0x7f80e0078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:39.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.473+0000 7f80ffe00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80e0076630 0x7f80e0078af0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81001f5880 tx=0x7f80ec0069b0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:39.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.476+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f80f0061560 con 0x7f80e4004d90 2024-07-26T06:19:39.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.689+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81000b34f0 con 0x7f80e4004d90 2024-07-26T06:19:39.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.690+0000 7f80fcc00640 1 -- 172.21.15.163:0/1947437390 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+13050 (secure 0 0 0) 0x7f80f0060e60 con 0x7f80e4004d90 2024-07-26T06:19:39.691 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:39.691 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":104,"fsid":"a1639b17-d73f-4cae-9471-025ba01f9572","created":"2024-07-26T06:13:08.235059+0000","modified":"2024-07-26T06:19:28.773999+0000","last_up_change":"2024-07-26T06:13:21.818251+0000","last_in_change":"2024-07-26T06:13:12.857980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:13:21.677101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72caccc6-2cb3-4d5d-bb11-9eacb422aeec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6801","nonce":337444050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6803","nonce":337444050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6807","nonce":337444050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":337444050},{"type":"v1","addr":"172.21.15.163:6805","nonce":337444050}]},"public_addr":"172.21.15.163:6801/337444050","cluster_addr":"172.21.15.163:6803/337444050","heartbeat_back_addr":"172.21.15.163:6807/337444050","heartbeat_front_addr":"172.21.15.163:6805/337444050","state":["exists","up"]},{"osd":1,"uuid":"69942df7-d9b8-4051-b4ec-422cf7beee9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6817","nonce":2784916277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6819","nonce":2784916277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6823","nonce":2784916277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2784916277},{"type":"v1","addr":"172.21.15.163:6821","nonce":2784916277}]},"public_addr":"172.21.15.163:6817/2784916277","cluster_addr":"172.21.15.163:6819/2784916277","heartbeat_back_addr":"172.21.15.163:6823/2784916277","heartbeat_front_addr":"172.21.15.163:6821/2784916277","state":["exists","up"]},{"osd":2,"uuid":"ed5f78dd-777d-4cb0-a6d7-5f6442deba3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6809","nonce":1222541827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6811","nonce":1222541827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6815","nonce":1222541827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":1222541827},{"type":"v1","addr":"172.21.15.163:6813","nonce":1222541827}]},"public_addr":"172.21.15.163:6809/1222541827","cluster_addr":"172.21.15.163:6811/1222541827","heartbeat_back_addr":"172.21.15.163:6815/1222541827","heartbeat_front_addr":"172.21.15.163:6813/1222541827","state":["exists","up"]},{"osd":3,"uuid":"1f2fe985-5805-4253-bd18-d1b35e1f5dcf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6825","nonce":136015621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6827","nonce":136015621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6831","nonce":136015621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":136015621},{"type":"v1","addr":"172.21.15.163:6829","nonce":136015621}]},"public_addr":"172.21.15.163:6825/136015621","cluster_addr":"172.21.15.163:6827/136015621","heartbeat_back_addr":"172.21.15.163:6831/136015621","heartbeat_front_addr":"172.21.15.163:6829/136015621","state":["exists","up"]},{"osd":4,"uuid":"17f99c8c-2d53-49ac-b648-1cb8f6f36466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6827","nonce":3692835080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6829","nonce":3692835080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6833","nonce":3692835080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3692835080},{"type":"v1","addr":"172.21.15.164:6831","nonce":3692835080}]},"public_addr":"172.21.15.164:6827/3692835080","cluster_addr":"172.21.15.164:6829/3692835080","heartbeat_back_addr":"172.21.15.164:6833/3692835080","heartbeat_front_addr":"172.21.15.164:6831/3692835080","state":["exists","up"]},{"osd":5,"uuid":"edacd1f8-feda-4c3e-b41b-7ca4420d418e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6801","nonce":3774389112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6803","nonce":3774389112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6807","nonce":3774389112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3774389112},{"type":"v1","addr":"172.21.15.164:6805","nonce":3774389112}]},"public_addr":"172.21.15.164:6801/3774389112","cluster_addr":"172.21.15.164:6803/3774389112","heartbeat_back_addr":"172.21.15.164:6807/3774389112","heartbeat_front_addr":"172.21.15.164:6805/3774389112","state":["exists","up"]},{"osd":6,"uuid":"1ddea884-310a-4d70-8dbe-d2c9f30f2436","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6809","nonce":2876752734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6811","nonce":2876752734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6815","nonce":2876752734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2876752734},{"type":"v1","addr":"172.21.15.164:6813","nonce":2876752734}]},"public_addr":"172.21.15.164:6809/2876752734","cluster_addr":"172.21.15.164:6811/2876752734","heartbeat_back_addr":"172.21.15.164:6815/2876752734","heartbeat_front_addr":"172.21.15.164:6813/2876752734","state":["exists","up"]},{"osd":7,"uuid":"0d239eae-64f6-47e6-a731-2e29aceb30ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6819","nonce":734207166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6821","nonce":734207166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6825","nonce":734207166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":734207166},{"type":"v1","addr":"172.21.15.164:6823","nonce":734207166}]},"public_addr":"172.21.15.164:6819/734207166","cluster_addr":"172.21.15.164:6821/734207166","heartbeat_back_addr":"172.21.15.164:6825/734207166","heartbeat_front_addr":"172.21.15.164:6823/734207166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.435248+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.252362+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.494304+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.303612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.236586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.423696+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.505769+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:13:18.493493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:6834/1667488712":"2024-07-27T06:19:25.762658+0000","172.21.15.163:6832/1667488712":"2024-07-27T06:19:25.762658+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:19:39.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80e0076630 msgr2=0x7f80e0078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80e0076630 0x7f80e0078af0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81001f5880 tx=0x7f80ec0069b0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 msgr2=0x7f81001f7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:39.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f81001f7220 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e540 tx=0x7f80f000ea00 comp rx=0 tx=0).stop 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 shutdown_connections 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f80e0076630 0x7f80e0078af0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000cc590 0x7f81000ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81001f5460 0x7f810014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 --2- 172.21.15.163:0/1947437390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80e4004d90 0x7f81001f7220 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 >> 172.21.15.163:0/1947437390 conn(0x7f81000b0e30 msgr2=0x7f81000b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:39.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.693+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 shutdown_connections 2024-07-26T06:19:39.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:39.694+0000 7f8105e00640 1 -- 172.21.15.163:0/1947437390 wait complete. 2024-07-26T06:19:40.715 INFO:tasks.ceph:Scrubbing osd.0 2024-07-26T06:19:40.716 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:40.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.937+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2407649230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4000ce4a0 msgr2=0x7fb400071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:40.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.937+0000 7fb3ffe00640 1 --2- 172.21.15.163:0/2407649230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4000ce4a0 0x7fb400071530 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8005fd0 tx=0x7fb3e802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:19:40.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.937+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2407649230 shutdown_connections 2024-07-26T06:19:40.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.937+0000 7fb3ffe00640 1 --2- 172.21.15.163:0/2407649230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4000ce4a0 0x7fb400071530 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:40.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.937+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2407649230 >> 172.21.15.163:0/2407649230 conn(0x7fb4000b0e30 msgr2=0x7fb4000b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:40.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.937+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2407649230 shutdown_connections 2024-07-26T06:19:40.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.938+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2407649230 wait complete. 2024-07-26T06:19:40.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.939+0000 7fb3ffe00640 1 Processor -- start 2024-07-26T06:19:40.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.939+0000 7fb3ffe00640 1 -- start start 2024-07-26T06:19:40.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.940+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb400071a70 0x7fb40014a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:40.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.940+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4000ce4a0 0x7fb4000b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:40.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.940+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4001556f0 0x7fb4001f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:40.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.940+0000 7fb3ffe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb4000b41f0 con 0x7fb400071a70 2024-07-26T06:19:40.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.940+0000 7fb3ffe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb4000b4070 con 0x7fb4000ce4a0 2024-07-26T06:19:40.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.940+0000 7fb3ffe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb4000b4370 con 0x7fb4001556f0 2024-07-26T06:19:40.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb400071a70 0x7fb40014a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:40.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb400071a70 0x7fb40014a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34218/0 (socket says 172.21.15.163:34218) 2024-07-26T06:19:40.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 -- 172.21.15.163:0/2566813664 learned_addr learned my addr 172.21.15.163:0/2566813664 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3f7e00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4000ce4a0 0x7fb4000b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3ff400640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4001556f0 0x7fb4001f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 -- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4001556f0 msgr2=0x7fb4001f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4001556f0 0x7fb4001f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 -- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4000ce4a0 msgr2=0x7fb4000b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4000ce4a0 0x7fb4000b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:40.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 -- 172.21.15.163:0/2566813664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4000ab800 con 0x7fb400071a70 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3f7e00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4000ce4a0 0x7fb4000b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.941+0000 7fb3fea00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb400071a70 0x7fb40014a7d0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c3d0 tx=0x7fb3f000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.942+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0017070 con 0x7fb400071a70 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.942+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3f00053c0 con 0x7fb400071a70 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.942+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0003850 con 0x7fb400071a70 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.942+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4000abae0 con 0x7fb400071a70 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.942+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4000ac0a0 con 0x7fb400071a70 2024-07-26T06:19:40.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.943+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb400071530 con 0x7fb400071a70 2024-07-26T06:19:40.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.944+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fb3f001a070 con 0x7fb400071a70 2024-07-26T06:19:40.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.945+0000 7fb3f7400640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb3dc076630 0x7fb3dc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:40.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.945+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fb3f0096e30 con 0x7fb400071a70 2024-07-26T06:19:40.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.945+0000 7fb3f7400640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7fb3dc07c4f0 0x7fb3dc07e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:40.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.945+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 --> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3dc07efc0 con 0x7fb3dc07c4f0 2024-07-26T06:19:40.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.945+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fb3f009c440 con 0x7fb400071a70 2024-07-26T06:19:40.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.946+0000 7fb3f7e00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb3dc076630 0x7fb3dc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:40.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.946+0000 7fb3ff400640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7fb3dc07c4f0 0x7fb3dc07e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:40.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.946+0000 7fb3ff400640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7fb3dc07c4f0 0x7fb3dc07e910 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:40.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.946+0000 7fb3f7e00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb3dc076630 0x7fb3dc078af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8033000 tx=0x7fb3e8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:40.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.947+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== osd.0 v2:172.21.15.163:6800/337444050 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3dc07efc0 con 0x7fb3dc07c4f0 2024-07-26T06:19:40.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.962+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 --> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb400154ea0 con 0x7fb3dc07c4f0 2024-07-26T06:19:40.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.962+0000 7fb3f7400640 1 -- 172.21.15.163:0/2566813664 <== osd.0 v2:172.21.15.163:6800/337444050 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb400154ea0 con 0x7fb3dc07c4f0 2024-07-26T06:19:40.963 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:40.963 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:40.963 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7fb3dc07c4f0 msgr2=0x7fb3dc07e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:40.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6800/337444050,v1:172.21.15.163:6801/337444050] conn(0x7fb3dc07c4f0 0x7fb3dc07e910 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb3dc076630 msgr2=0x7fb3dc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fb3dc076630 0x7fb3dc078af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8033000 tx=0x7fb3e8004730 comp rx=0 tx=0).stop 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb400071a70 msgr2=0x7fb40014a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 --2- 172.21.15.163:0/2566813664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb400071a70 0x7fb40014a7d0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c3d0 tx=0x7fb3f000c890 comp rx=0 tx=0).stop 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ff400640 1 -- 172.21.15.163:0/2566813664 reap_dead start 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 shutdown_connections 2024-07-26T06:19:40.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 >> 172.21.15.163:0/2566813664 conn(0x7fb4000b0e30 msgr2=0x7fb400151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:40.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 shutdown_connections 2024-07-26T06:19:40.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:40.963+0000 7fb3ffe00640 1 -- 172.21.15.163:0/2566813664 wait complete. 2024-07-26T06:19:40.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-07-26T06:19:41.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 -- 172.21.15.163:0/1963844059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780b4420 msgr2=0x7ff4780b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 --2- 172.21.15.163:0/1963844059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780b4420 0x7ff4780b2130 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff460005fd0 tx=0x7ff46002a5d0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 -- 172.21.15.163:0/1963844059 shutdown_connections 2024-07-26T06:19:41.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 --2- 172.21.15.163:0/1963844059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780b4420 0x7ff4780b2130 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 -- 172.21.15.163:0/1963844059 >> 172.21.15.163:0/1963844059 conn(0x7ff4780b0e30 msgr2=0x7ff4780b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:41.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 -- 172.21.15.163:0/1963844059 shutdown_connections 2024-07-26T06:19:41.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.184+0000 7ff477e00640 1 -- 172.21.15.163:0/1963844059 wait complete. 2024-07-26T06:19:41.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.186+0000 7ff477e00640 1 Processor -- start 2024-07-26T06:19:41.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.186+0000 7ff477e00640 1 -- start start 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff477e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4780b4420 0x7ff478153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff477e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 0x7ff47814c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff477e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47814e000 0x7ff478149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff477e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff4780b2d80 con 0x7ff4780cdc40 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff477e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff47806f500 con 0x7ff4780b4420 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff477e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4780b2f00 con 0x7ff47814e000 2024-07-26T06:19:41.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 0x7ff47814c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 0x7ff47814c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34232/0 (socket says 172.21.15.163:34232) 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.187+0000 7ff46fe00640 1 -- 172.21.15.163:0/1304418105 learned_addr learned my addr 172.21.15.163:0/1304418105 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff476a00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4780b4420 0x7ff478153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff477400640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47814e000 0x7ff478149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46fe00640 1 -- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47814e000 msgr2=0x7ff478149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46fe00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47814e000 0x7ff478149050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46fe00640 1 -- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4780b4420 msgr2=0x7ff478153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46fe00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4780b4420 0x7ff478153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46fe00640 1 -- 172.21.15.163:0/1304418105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff478153550 con 0x7ff4780cdc40 2024-07-26T06:19:41.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46fe00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 0x7ff47814c660 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff460009850 tx=0x7ff4600038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.188+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46003f070 con 0x7ff4780cdc40 2024-07-26T06:19:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.189+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4600047a0 con 0x7ff4780cdc40 2024-07-26T06:19:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.189+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47814cba0 con 0x7ff4780cdc40 2024-07-26T06:19:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.189+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460035030 con 0x7ff4780cdc40 2024-07-26T06:19:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.189+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4780ade10 con 0x7ff4780cdc40 2024-07-26T06:19:41.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.190+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4780cd530 con 0x7ff4780cdc40 2024-07-26T06:19:41.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.191+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7ff460042070 con 0x7ff4780cdc40 2024-07-26T06:19:41.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.192+0000 7ff46ea00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff454076680 0x7ff454078b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.192+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7ff4600b8670 con 0x7ff4780cdc40 2024-07-26T06:19:41.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.192+0000 7ff476a00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff454076680 0x7ff454078b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.193+0000 7ff476a00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff454076680 0x7ff454078b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff4780cdbe0 tx=0x7ff468009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.197+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff47814cba0 con 0x7ff4780cdc40 2024-07-26T06:19:41.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.407+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7ff4780ad8e0 con 0x7ff454076680 2024-07-26T06:19:41.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.408+0000 7ff46ea00640 1 -- 172.21.15.163:0/1304418105 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff4780ad8e0 con 0x7ff454076680 2024-07-26T06:19:41.460 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 0 to deep-scrub 2024-07-26T06:19:41.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff454076680 msgr2=0x7ff454078b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff454076680 0x7ff454078b40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff4780cdbe0 tx=0x7ff468009040 comp rx=0 tx=0).stop 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 msgr2=0x7ff47814c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 0x7ff47814c660 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff460009850 tx=0x7ff4600038a0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 shutdown_connections 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff454076680 0x7ff454078b40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47814e000 0x7ff478149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4780cdc40 0x7ff47814c660 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 --2- 172.21.15.163:0/1304418105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4780b4420 0x7ff478153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 >> 172.21.15.163:0/1304418105 conn(0x7ff4780b0e30 msgr2=0x7ff478159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:41.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 shutdown_connections 2024-07-26T06:19:41.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.411+0000 7ff477e00640 1 -- 172.21.15.163:0/1304418105 wait complete. 2024-07-26T06:19:41.462 INFO:tasks.ceph:Scrubbing osd.1 2024-07-26T06:19:41.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:41.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e2800640 1 -- 172.21.15.163:0/1227366240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1c4004800 msgr2=0x7fe1c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e2800640 1 --2- 172.21.15.163:0/1227366240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1c4004800 0x7fe1c4004c80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc160e50 tx=0x7fe1c4005830 comp rx=0 tx=0).stop 2024-07-26T06:19:41.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e1e00640 1 -- 172.21.15.163:0/1227366240 reap_dead start 2024-07-26T06:19:41.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e2800640 1 -- 172.21.15.163:0/1227366240 shutdown_connections 2024-07-26T06:19:41.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e2800640 1 -- 172.21.15.163:0/1227366240 >> 172.21.15.163:0/1227366240 conn(0x7fe1dc0cfda0 msgr2=0x7fe1dc0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:41.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e2800640 1 -- 172.21.15.163:0/1227366240 shutdown_connections 2024-07-26T06:19:41.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.683+0000 7fe1e2800640 1 -- 172.21.15.163:0/1227366240 wait complete. 2024-07-26T06:19:41.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.685+0000 7fe1e2800640 1 Processor -- start 2024-07-26T06:19:41.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.685+0000 7fe1e2800640 1 -- start start 2024-07-26T06:19:41.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e2800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1c4004800 0x7fe1dc0c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e2800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1dc14d7f0 0x7fe1dc155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e2800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1dc15bb40 0x7fe1dc15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e2800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc0ce470 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e2800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc0ce2f0 con 0x7fe1dc15bb40 2024-07-26T06:19:41.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e2800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe1dc0ce5f0 con 0x7fe1c4004800 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1dc14d7f0 0x7fe1dc155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1dc14d7f0 0x7fe1dc155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34244/0 (socket says 172.21.15.163:34244) 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 -- 172.21.15.163:0/3587271062 learned_addr learned my addr 172.21.15.163:0/3587271062 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e1400640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1c4004800 0x7fe1dc0c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 -- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1c4004800 msgr2=0x7fe1dc0c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1c4004800 0x7fe1dc0c6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e1e00640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1dc15bb40 0x7fe1dc15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 -- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1dc15bb40 msgr2=0x7fe1dc15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1dc15bb40 0x7fe1dc15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e0a00640 1 -- 172.21.15.163:0/3587271062 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc0c7430 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.686+0000 7fe1e1400640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1c4004800 0x7fe1dc0c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:41.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.687+0000 7fe1e0a00640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1dc14d7f0 0x7fe1dc155130 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009e20 tx=0x7fe1cc033bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.687+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc038040 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.687+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1cc02bbe0 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.687+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc004b20 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.687+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1dc0c7710 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.687+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1dc155670 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.688+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe1dc0c8560 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.689+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fe1cc045020 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1d9600640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe1c0076630 0x7fe1c0078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fe1cc0b8610 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1e1400640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe1c0076630 0x7fe1c0078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1d9600640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fe1c007c4f0 0x7fe1c007e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 --> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1c007efc0 con 0x7fe1c007c4f0 2024-07-26T06:19:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fe1cc0bf050 con 0x7fe1dc14d7f0 2024-07-26T06:19:41.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.691+0000 7fe1e1e00640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fe1c007c4f0 0x7fe1c007e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.692+0000 7fe1e1400640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe1c0076630 0x7fe1c0078af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009ae0 tx=0x7fe1d00095c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.692+0000 7fe1e1e00640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fe1c007c4f0 0x7fe1c007e910 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.693+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== osd.1 v2:172.21.15.163:6816/2784916277 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe1c007efc0 con 0x7fe1c007c4f0 2024-07-26T06:19:41.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.707+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 --> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe1dc0b5120 con 0x7fe1c007c4f0 2024-07-26T06:19:41.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.708+0000 7fe1d9600640 1 -- 172.21.15.163:0/3587271062 <== osd.1 v2:172.21.15.163:6816/2784916277 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe1dc0b5120 con 0x7fe1c007c4f0 2024-07-26T06:19:41.709 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:41.709 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:41.709 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:41.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.708+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fe1c007c4f0 msgr2=0x7fe1c007e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.708+0000 7fe1e2800640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6816/2784916277,v1:172.21.15.163:6817/2784916277] conn(0x7fe1c007c4f0 0x7fe1c007e910 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe1c0076630 msgr2=0x7fe1c0078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe1c0076630 0x7fe1c0078af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009ae0 tx=0x7fe1d00095c0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1dc14d7f0 msgr2=0x7fe1dc155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 --2- 172.21.15.163:0/3587271062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1dc14d7f0 0x7fe1dc155130 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009e20 tx=0x7fe1cc033bc0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e1e00640 1 -- 172.21.15.163:0/3587271062 reap_dead start 2024-07-26T06:19:41.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 shutdown_connections 2024-07-26T06:19:41.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 >> 172.21.15.163:0/3587271062 conn(0x7fe1dc0cfda0 msgr2=0x7fe1dc158620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:41.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 shutdown_connections 2024-07-26T06:19:41.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.709+0000 7fe1e2800640 1 -- 172.21.15.163:0/3587271062 wait complete. 2024-07-26T06:19:41.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-07-26T06:19:41.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.917+0000 7f4d49000640 1 -- 172.21.15.163:0/3838554760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d44073820 msgr2=0x7f4d44073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.917+0000 7f4d49000640 1 --2- 172.21.15.163:0/3838554760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d44073820 0x7f4d44073cb0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009830 tx=0x7f4d3c02a480 comp rx=0 tx=0).stop 2024-07-26T06:19:41.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.917+0000 7f4d49000640 1 -- 172.21.15.163:0/3838554760 shutdown_connections 2024-07-26T06:19:41.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.917+0000 7f4d49000640 1 --2- 172.21.15.163:0/3838554760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d44073820 0x7f4d44073cb0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.917+0000 7f4d49000640 1 --2- 172.21.15.163:0/3838554760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440ae870 0x7f4d440b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.917+0000 7f4d49000640 1 -- 172.21.15.163:0/3838554760 >> 172.21.15.163:0/3838554760 conn(0x7f4d440b0e30 msgr2=0x7f4d440b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:41.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.918+0000 7f4d49000640 1 -- 172.21.15.163:0/3838554760 shutdown_connections 2024-07-26T06:19:41.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.918+0000 7f4d49000640 1 -- 172.21.15.163:0/3838554760 wait complete. 2024-07-26T06:19:41.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.919+0000 7f4d49000640 1 Processor -- start 2024-07-26T06:19:41.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.919+0000 7f4d49000640 1 -- start start 2024-07-26T06:19:41.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.920+0000 7f4d49000640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d44073820 0x7f4d4414d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.920+0000 7f4d49000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 0x7f4d441f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.920+0000 7f4d49000640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440b3bd0 0x7f4d440ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.920+0000 7f4d49000640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4415a1d0 con 0x7f4d440ae870 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.920+0000 7f4d49000640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4d44073380 con 0x7f4d44073820 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.920+0000 7f4d49000640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4415a350 con 0x7f4d440b3bd0 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 0x7f4d441f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 0x7f4d441f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34248/0 (socket says 172.21.15.163:34248) 2024-07-26T06:19:41.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 -- 172.21.15.163:0/1844685469 learned_addr learned my addr 172.21.15.163:0/1844685469 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d43400640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d44073820 0x7f4d4414d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d43e00640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440b3bd0 0x7f4d440ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 -- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440b3bd0 msgr2=0x7f4d440ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440b3bd0 0x7f4d440ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 -- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d44073820 msgr2=0x7f4d4414d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d44073820 0x7f4d4414d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 -- 172.21.15.163:0/1844685469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d44154600 con 0x7f4d440ae870 2024-07-26T06:19:41.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d43e00640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440b3bd0 0x7f4d440ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:41.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.921+0000 7f4d42a00640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 0x7f4d441f7140 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000eae0 tx=0x7f4d3000c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.922+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3000ce50 con 0x7f4d440ae870 2024-07-26T06:19:41.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.922+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d30011560 con 0x7f4d440ae870 2024-07-26T06:19:41.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.922+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30011990 con 0x7f4d440ae870 2024-07-26T06:19:41.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.922+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d441f7ab0 con 0x7f4d440ae870 2024-07-26T06:19:41.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.922+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4414e640 con 0x7f4d440ae870 2024-07-26T06:19:41.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.923+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d441f7fd0 con 0x7f4d440ae870 2024-07-26T06:19:41.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.924+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f4d30011b30 con 0x7f4d440ae870 2024-07-26T06:19:41.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.925+0000 7f4d3b400640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d240767d0 0x7f4d24078c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:41.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.925+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f4d300986a0 con 0x7f4d440ae870 2024-07-26T06:19:41.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.925+0000 7f4d43400640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d240767d0 0x7f4d24078c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:41.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.926+0000 7f4d43400640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d240767d0 0x7f4d24078c90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4d440af340 tx=0x7f4d34009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:41.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:41.930+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d441f7fd0 con 0x7f4d440ae870 2024-07-26T06:19:42.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.135+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f4d44074420 con 0x7f4d240767d0 2024-07-26T06:19:42.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.136+0000 7f4d3b400640 1 -- 172.21.15.163:0/1844685469 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4d44074420 con 0x7f4d240767d0 2024-07-26T06:19:42.137 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 1 to deep-scrub 2024-07-26T06:19:42.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.138+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d240767d0 msgr2=0x7f4d24078c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.138+0000 7f4d49000640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d240767d0 0x7f4d24078c90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4d440af340 tx=0x7f4d34009040 comp rx=0 tx=0).stop 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.138+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 msgr2=0x7f4d441f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.138+0000 7f4d49000640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 0x7f4d441f7140 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000eae0 tx=0x7f4d3000c270 comp rx=0 tx=0).stop 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 shutdown_connections 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4d240767d0 0x7f4d24078c90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d440b3bd0 0x7f4d440ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d440ae870 0x7f4d441f7140 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 --2- 172.21.15.163:0/1844685469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d44073820 0x7f4d4414d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 >> 172.21.15.163:0/1844685469 conn(0x7f4d440b0e30 msgr2=0x7f4d44156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 shutdown_connections 2024-07-26T06:19:42.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.139+0000 7f4d49000640 1 -- 172.21.15.163:0/1844685469 wait complete. 2024-07-26T06:19:42.157 INFO:tasks.ceph:Scrubbing osd.2 2024-07-26T06:19:42.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:42.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 -- 172.21.15.163:0/2316829039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a00ce4a0 msgr2=0x7fe5a0071530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 --2- 172.21.15.163:0/2316829039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a00ce4a0 0x7fe5a0071530 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fe588009830 tx=0x7fe58802a510 comp rx=0 tx=0).stop 2024-07-26T06:19:42.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 -- 172.21.15.163:0/2316829039 shutdown_connections 2024-07-26T06:19:42.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 --2- 172.21.15.163:0/2316829039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a0071a70 0x7fe5a00b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 --2- 172.21.15.163:0/2316829039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a00ce4a0 0x7fe5a0071530 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 -- 172.21.15.163:0/2316829039 >> 172.21.15.163:0/2316829039 conn(0x7fe5a00b0e30 msgr2=0x7fe5a00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:42.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 -- 172.21.15.163:0/2316829039 shutdown_connections 2024-07-26T06:19:42.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.360+0000 7fe5a4e00640 1 -- 172.21.15.163:0/2316829039 wait complete. 2024-07-26T06:19:42.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.362+0000 7fe5a4e00640 1 Processor -- start 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.362+0000 7fe5a4e00640 1 -- start start 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe5a4e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0071a70 0x7fe5a00adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe5a4e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a00ce4a0 0x7fe5a015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe5a4e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5a01556f0 0x7fe5a015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe5a4e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a00b3fe0 con 0x7fe5a0071a70 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe5a4e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a00b3e60 con 0x7fe5a01556f0 2024-07-26T06:19:42.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe5a4e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe5a00b4160 con 0x7fe5a00ce4a0 2024-07-26T06:19:42.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.363+0000 7fe59f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0071a70 0x7fe5a00adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0071a70 0x7fe5a00adae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34256/0 (socket says 172.21.15.163:34256) 2024-07-26T06:19:42.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 -- 172.21.15.163:0/3592337077 learned_addr learned my addr 172.21.15.163:0/3592337077 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:42.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 -- 172.21.15.163:0/3592337077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a00ce4a0 msgr2=0x7fe5a015a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a00ce4a0 0x7fe5a015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 -- 172.21.15.163:0/3592337077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5a01556f0 msgr2=0x7fe5a015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5a01556f0 0x7fe5a015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 -- 172.21.15.163:0/3592337077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a00ae4f0 con 0x7fe5a0071a70 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe59f400640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0071a70 0x7fe5a00adae0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fe59000c3e0 tx=0x7fe59000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590011790 con 0x7fe5a0071a70 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe590017070 con 0x7fe5a0071a70 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.364+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590011ce0 con 0x7fe5a0071a70 2024-07-26T06:19:42.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.365+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a01e5530 con 0x7fe5a0071a70 2024-07-26T06:19:42.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.365+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a01e5af0 con 0x7fe5a0071a70 2024-07-26T06:19:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.366+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5a00b46f0 con 0x7fe5a0071a70 2024-07-26T06:19:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.367+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fe59001c460 con 0x7fe5a0071a70 2024-07-26T06:19:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.368+0000 7fe597400640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe57c076630 0x7fe57c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.368+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fe5900977b0 con 0x7fe5a0071a70 2024-07-26T06:19:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.368+0000 7fe597e00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe57c076630 0x7fe57c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.368+0000 7fe597400640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7fe57c07c4f0 0x7fe57c07e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.369+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 --> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe57c07efc0 con 0x7fe57c07c4f0 2024-07-26T06:19:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.369+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7fe59009e050 con 0x7fe5a0071a70 2024-07-26T06:19:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.369+0000 7fe59fe00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7fe57c07c4f0 0x7fe57c07e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.369+0000 7fe597e00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe57c076630 0x7fe57c078af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe588034000 tx=0x7fe5880046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.369+0000 7fe59fe00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7fe57c07c4f0 0x7fe57c07e910 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.370+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== osd.2 v2:172.21.15.163:6808/1222541827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe57c07efc0 con 0x7fe57c07c4f0 2024-07-26T06:19:42.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.385+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 --> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe5a00cb520 con 0x7fe57c07c4f0 2024-07-26T06:19:42.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.385+0000 7fe597400640 1 -- 172.21.15.163:0/3592337077 <== osd.2 v2:172.21.15.163:6808/1222541827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe5a00cb520 con 0x7fe57c07c4f0 2024-07-26T06:19:42.387 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:42.387 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:42.387 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:42.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.385+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7fe57c07c4f0 msgr2=0x7fe57c07e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.385+0000 7fe5a4e00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6808/1222541827,v1:172.21.15.163:6809/1222541827] conn(0x7fe57c07c4f0 0x7fe57c07e910 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe57c076630 msgr2=0x7fe57c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fe57c076630 0x7fe57c078af0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe588034000 tx=0x7fe5880046c0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0071a70 msgr2=0x7fe5a00adae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 --2- 172.21.15.163:0/3592337077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5a0071a70 0x7fe5a00adae0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fe59000c3e0 tx=0x7fe59000c8a0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe59fe00640 1 -- 172.21.15.163:0/3592337077 reap_dead start 2024-07-26T06:19:42.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 shutdown_connections 2024-07-26T06:19:42.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 >> 172.21.15.163:0/3592337077 conn(0x7fe5a00b0e30 msgr2=0x7fe5a00b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:42.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 shutdown_connections 2024-07-26T06:19:42.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.386+0000 7fe5a4e00640 1 -- 172.21.15.163:0/3592337077 wait complete. 2024-07-26T06:19:42.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-07-26T06:19:42.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 -- 172.21.15.163:0/280669242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18140b3b80 msgr2=0x7f18140ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 --2- 172.21.15.163:0/280669242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18140b3b80 0x7f18140ce500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009830 tx=0x7f17fc02a510 comp rx=0 tx=0).stop 2024-07-26T06:19:42.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 -- 172.21.15.163:0/280669242 shutdown_connections 2024-07-26T06:19:42.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 --2- 172.21.15.163:0/280669242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18140b3b80 0x7f18140ce500 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 --2- 172.21.15.163:0/280669242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1814144c40 0x7f18140cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 -- 172.21.15.163:0/280669242 >> 172.21.15.163:0/280669242 conn(0x7f18140b3060 msgr2=0x7f18140b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:42.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 -- 172.21.15.163:0/280669242 shutdown_connections 2024-07-26T06:19:42.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.607+0000 7f1813e00640 1 -- 172.21.15.163:0/280669242 wait complete. 2024-07-26T06:19:42.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.609+0000 7f1813e00640 1 Processor -- start 2024-07-26T06:19:42.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.609+0000 7f1813e00640 1 -- start start 2024-07-26T06:19:42.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.610+0000 7f1813e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1814071990 0x7f18140cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.610+0000 7f1813e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 0x7f181406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.610+0000 7f1813e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1814144c40 0x7f181406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.610+0000 7f1813e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f18140d06f0 con 0x7f18140b3b80 2024-07-26T06:19:42.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.610+0000 7f1813e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f18140713a0 con 0x7f1814071990 2024-07-26T06:19:42.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.610+0000 7f1813e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f18140d0870 con 0x7f1814144c40 2024-07-26T06:19:42.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 0x7f181406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 0x7f181406e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34268/0 (socket says 172.21.15.163:34268) 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 -- 172.21.15.163:0/448344258 learned_addr learned my addr 172.21.15.163:0/448344258 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812a00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1814071990 0x7f18140cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1813400640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1814144c40 0x7f181406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 -- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1814144c40 msgr2=0x7f181406eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1814144c40 0x7f181406eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 -- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1814071990 msgr2=0x7f18140cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1814071990 0x7f18140cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 -- 172.21.15.163:0/448344258 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1814154760 con 0x7f18140b3b80 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.611+0000 7f1812000640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 0x7f181406e580 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f17fc000c00 tx=0x7f17fc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.612+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc040070 con 0x7f18140b3b80 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.612+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17fc0038e0 con 0x7f18140b3b80 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.612+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc003d30 con 0x7f18140b3b80 2024-07-26T06:19:42.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.612+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1814154a40 con 0x7f18140b3b80 2024-07-26T06:19:42.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.612+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1814155000 con 0x7f18140b3b80 2024-07-26T06:19:42.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.613+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18140d0d00 con 0x7f18140b3b80 2024-07-26T06:19:42.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.614+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f17fc039040 con 0x7f18140b3b80 2024-07-26T06:19:42.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.615+0000 7f180aa00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f17f0076630 0x7f17f0078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:42.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.615+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f17fc0b9070 con 0x7f18140b3b80 2024-07-26T06:19:42.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.616+0000 7f1812a00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f17f0076630 0x7f17f0078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:42.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.616+0000 7f1812a00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f17f0076630 0x7f17f0078af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1804005fd0 tx=0x7f18040095c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:42.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.620+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f17fc082800 con 0x7f18140b3b80 2024-07-26T06:19:42.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.823+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f18140b2600 con 0x7f17f0076630 2024-07-26T06:19:42.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.824+0000 7f180aa00640 1 -- 172.21.15.163:0/448344258 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f18140b2600 con 0x7f17f0076630 2024-07-26T06:19:42.825 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 2 to deep-scrub 2024-07-26T06:19:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.826+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f17f0076630 msgr2=0x7f17f0078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.826+0000 7f1813e00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f17f0076630 0x7f17f0078af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1804005fd0 tx=0x7f18040095c0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 msgr2=0x7f181406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 0x7f181406e580 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f17fc000c00 tx=0x7f17fc032c10 comp rx=0 tx=0).stop 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 shutdown_connections 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f17f0076630 0x7f17f0078af0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1814144c40 0x7f181406eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18140b3b80 0x7f181406e580 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 --2- 172.21.15.163:0/448344258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1814071990 0x7f18140cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 >> 172.21.15.163:0/448344258 conn(0x7f18140b3060 msgr2=0x7f18140b5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 shutdown_connections 2024-07-26T06:19:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:42.827+0000 7f1813e00640 1 -- 172.21.15.163:0/448344258 wait complete. 2024-07-26T06:19:42.848 INFO:tasks.ceph:Scrubbing osd.3 2024-07-26T06:19:42.849 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:43.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.056+0000 7f6411a00640 1 -- 172.21.15.163:0/3002994012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63f0005210 msgr2=0x7f63f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.056+0000 7f6411a00640 1 --2- 172.21.15.163:0/3002994012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63f0005210 0x7f63f000b9c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f640400c560 tx=0x7f63f0005830 comp rx=0 tx=0).stop 2024-07-26T06:19:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.056+0000 7f6411a00640 1 -- 172.21.15.163:0/3002994012 shutdown_connections 2024-07-26T06:19:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.056+0000 7f6411a00640 1 --2- 172.21.15.163:0/3002994012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63f0005210 0x7f63f000b9c0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.056+0000 7f6411a00640 1 -- 172.21.15.163:0/3002994012 >> 172.21.15.163:0/3002994012 conn(0x7f640c0b0e30 msgr2=0x7f640c0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.057+0000 7f6411a00640 1 -- 172.21.15.163:0/3002994012 shutdown_connections 2024-07-26T06:19:43.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.057+0000 7f6411a00640 1 -- 172.21.15.163:0/3002994012 wait complete. 2024-07-26T06:19:43.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.058+0000 7f6411a00640 1 Processor -- start 2024-07-26T06:19:43.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.058+0000 7f6411a00640 1 -- start start 2024-07-26T06:19:43.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6411a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f0005210 0x7f640c1f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6411a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f640c14c850 0x7f640c0cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6411a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f640c0cc590 0x7f640c14aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6411a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f640c0718c0 con 0x7f640c14c850 2024-07-26T06:19:43.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6411a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f640c071740 con 0x7f63f0005210 2024-07-26T06:19:43.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6411a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f640c0cdd90 con 0x7f640c0cc590 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.059+0000 7f6403400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f640c14c850 0x7f640c0cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f640c14c850 0x7f640c0cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34278/0 (socket says 172.21.15.163:34278) 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 -- 172.21.15.163:0/3170520489 learned_addr learned my addr 172.21.15.163:0/3170520489 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f640be00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f0005210 0x7f640c1f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6411000640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f640c0cc590 0x7f640c14aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 -- 172.21.15.163:0/3170520489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f640c0cc590 msgr2=0x7f640c14aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f640c0cc590 0x7f640c14aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 -- 172.21.15.163:0/3170520489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f0005210 msgr2=0x7f640c1f4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f0005210 0x7f640c1f4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 -- 172.21.15.163:0/3170520489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c14a730 con 0x7f640c14c850 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6403400640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f640c14c850 0x7f640c0cc050 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f63f400e930 tx=0x7f63f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.060+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f400cc60 con 0x7f640c14c850 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.061+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63f400ce00 con 0x7f640c14c850 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.061+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4004290 con 0x7f640c14c850 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.061+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640c1f4930 con 0x7f640c14c850 2024-07-26T06:19:43.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.061+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f640c1f4ef0 con 0x7f640c14c850 2024-07-26T06:19:43.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.062+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f640c15ec00 con 0x7f640c14c850 2024-07-26T06:19:43.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.063+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f63f40035f0 con 0x7f640c14c850 2024-07-26T06:19:43.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.064+0000 7f6408c00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f63e8076630 0x7f63e8078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.064+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f63f4098590 con 0x7f640c14c850 2024-07-26T06:19:43.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.064+0000 7f640be00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f63e8076630 0x7f63e8078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.064+0000 7f6408c00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7f63e807c4f0 0x7f63e807e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.064+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 --> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63e807efc0 con 0x7f63e807c4f0 2024-07-26T06:19:43.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.064+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f63f409f050 con 0x7f640c14c850 2024-07-26T06:19:43.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.065+0000 7f6411000640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7f63e807c4f0 0x7f63e807e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.065+0000 7f640be00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f63e8076630 0x7f63e8078af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f63fc001840 tx=0x7f63fc006b60 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.065+0000 7f6411000640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7f63e807c4f0 0x7f63e807e910 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.066+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== osd.3 v2:172.21.15.163:6824/136015621 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e807efc0 con 0x7f63e807c4f0 2024-07-26T06:19:43.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.080+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 --> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f640c0ced60 con 0x7f63e807c4f0 2024-07-26T06:19:43.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6408c00640 1 -- 172.21.15.163:0/3170520489 <== osd.3 v2:172.21.15.163:6824/136015621 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f640c0ced60 con 0x7f63e807c4f0 2024-07-26T06:19:43.082 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:43.083 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:43.083 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:43.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7f63e807c4f0 msgr2=0x7f63e807e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6824/136015621,v1:172.21.15.163:6825/136015621] conn(0x7f63e807c4f0 0x7f63e807e910 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f63e8076630 msgr2=0x7f63e8078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f63e8076630 0x7f63e8078af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f63fc001840 tx=0x7f63fc006b60 comp rx=0 tx=0).stop 2024-07-26T06:19:43.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f640c14c850 msgr2=0x7f640c0cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 --2- 172.21.15.163:0/3170520489 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f640c14c850 0x7f640c0cc050 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f63f400e930 tx=0x7f63f400edf0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411000640 1 -- 172.21.15.163:0/3170520489 reap_dead start 2024-07-26T06:19:43.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 shutdown_connections 2024-07-26T06:19:43.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.081+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 >> 172.21.15.163:0/3170520489 conn(0x7f640c0b0e30 msgr2=0x7f640c151c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:43.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.082+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 shutdown_connections 2024-07-26T06:19:43.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.082+0000 7f6411a00640 1 -- 172.21.15.163:0/3170520489 wait complete. 2024-07-26T06:19:43.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-07-26T06:19:43.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06c00640 1 -- 172.21.15.163:0/2639261794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbee4004000 msgr2=0x7fbee4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06c00640 1 --2- 172.21.15.163:0/2639261794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbee4004000 0x7fbee4004800 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbeec005fd0 tx=0x7fbee4005830 comp rx=0 tx=0).stop 2024-07-26T06:19:43.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06200640 1 -- 172.21.15.163:0/2639261794 reap_dead start 2024-07-26T06:19:43.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06c00640 1 -- 172.21.15.163:0/2639261794 shutdown_connections 2024-07-26T06:19:43.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06c00640 1 -- 172.21.15.163:0/2639261794 >> 172.21.15.163:0/2639261794 conn(0x7fbf000cfda0 msgr2=0x7fbf000d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:43.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06c00640 1 -- 172.21.15.163:0/2639261794 shutdown_connections 2024-07-26T06:19:43.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.300+0000 7fbf06c00640 1 -- 172.21.15.163:0/2639261794 wait complete. 2024-07-26T06:19:43.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.302+0000 7fbf06c00640 1 Processor -- start 2024-07-26T06:19:43.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.302+0000 7fbf06c00640 1 -- start start 2024-07-26T06:19:43.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf06c00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbee4004000 0x7fbf001f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf06c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 0x7fbf0015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf06c00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf0015c000 0x7fbf0015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf06c00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbf000b4da0 con 0x7fbf000cb000 2024-07-26T06:19:43.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf06c00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbf000b6720 con 0x7fbee4004000 2024-07-26T06:19:43.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf06c00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbf000b4f20 con 0x7fbf0015c000 2024-07-26T06:19:43.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf04e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 0x7fbf0015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf04e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 0x7fbf0015bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34300/0 (socket says 172.21.15.163:34300) 2024-07-26T06:19:43.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.303+0000 7fbf04e00640 1 -- 172.21.15.163:0/3508852785 learned_addr learned my addr 172.21.15.163:0/3508852785 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:43.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf05800640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbee4004000 0x7fbf001f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf04e00640 1 -- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf0015c000 msgr2=0x7fbf0015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf04e00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf0015c000 0x7fbf0015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf04e00640 1 -- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbee4004000 msgr2=0x7fbf001f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf04e00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbee4004000 0x7fbf001f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf04e00640 1 -- 172.21.15.163:0/3508852785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0015c540 con 0x7fbf000cb000 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf04e00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 0x7fbf0015bac0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbeec036040 tx=0x7fbeec033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec038040 con 0x7fbf000cb000 2024-07-26T06:19:43.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbeec02bc10 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004b20 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.304+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf000ca520 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.305+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0015dbd0 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.306+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf000b5480 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.307+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fbeec045020 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.308+0000 7fbefd600640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbee0076660 0x7fbee0078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.308+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fbeec0b8460 con 0x7fbf000cb000 2024-07-26T06:19:43.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.308+0000 7fbf05800640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbee0076660 0x7fbee0078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.309+0000 7fbf05800640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbee0076660 0x7fbee0078b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbef0000c00 tx=0x7fbef00069c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.312+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbeec081c90 con 0x7fbf000cb000 2024-07-26T06:19:43.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.516+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fbf000d2310 con 0x7fbee0076660 2024-07-26T06:19:43.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.517+0000 7fbefd600640 1 -- 172.21.15.163:0/3508852785 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbf000d2310 con 0x7fbee0076660 2024-07-26T06:19:43.518 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 3 to deep-scrub 2024-07-26T06:19:43.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbee0076660 msgr2=0x7fbee0078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbee0076660 0x7fbee0078b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbef0000c00 tx=0x7fbef00069c0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 msgr2=0x7fbf0015bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 0x7fbf0015bac0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbeec036040 tx=0x7fbeec033c30 comp rx=0 tx=0).stop 2024-07-26T06:19:43.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 shutdown_connections 2024-07-26T06:19:43.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fbee0076660 0x7fbee0078b20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf0015c000 0x7fbf0015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf000cb000 0x7fbf0015bac0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 --2- 172.21.15.163:0/3508852785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbee4004000 0x7fbf001f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 >> 172.21.15.163:0/3508852785 conn(0x7fbf000cfda0 msgr2=0x7fbf000d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:43.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 shutdown_connections 2024-07-26T06:19:43.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.520+0000 7fbf06c00640 1 -- 172.21.15.163:0/3508852785 wait complete. 2024-07-26T06:19:43.540 INFO:tasks.ceph:Scrubbing osd.4 2024-07-26T06:19:43.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:43.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2f200640 1 -- 172.21.15.163:0/342297953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 msgr2=0x7f4a14004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2f200640 1 --2- 172.21.15.163:0/342297953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 0x7f4a14004800 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800bb20 tx=0x7f4a14005830 comp rx=0 tx=0).stop 2024-07-26T06:19:43.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2e800640 1 -- 172.21.15.163:0/342297953 reap_dead start 2024-07-26T06:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2f200640 1 -- 172.21.15.163:0/342297953 shutdown_connections 2024-07-26T06:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2f200640 1 -- 172.21.15.163:0/342297953 >> 172.21.15.163:0/342297953 conn(0x7f4a300b0e30 msgr2=0x7f4a300b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2f200640 1 -- 172.21.15.163:0/342297953 shutdown_connections 2024-07-26T06:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.749+0000 7f4a2f200640 1 -- 172.21.15.163:0/342297953 wait complete. 2024-07-26T06:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.751+0000 7f4a2f200640 1 Processor -- start 2024-07-26T06:19:43.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.751+0000 7f4a2f200640 1 -- start start 2024-07-26T06:19:43.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2f200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 0x7f4a30143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2f200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a30145d40 0x7f4a301387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2f200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a300c58a0 0x7f4a300c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2f200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a300cc0d0 con 0x7f4a14004090 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2f200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4a300cbf50 con 0x7f4a300c58a0 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2f200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4a300b3e60 con 0x7f4a30145d40 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 0x7f4a30143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2de00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 0x7f4a30143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34332/0 (socket says 172.21.15.163:34332) 2024-07-26T06:19:43.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.752+0000 7f4a2de00640 1 -- 172.21.15.163:0/3143869185 learned_addr learned my addr 172.21.15.163:0/3143869185 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2e800640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a300c58a0 0x7f4a300c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a25400640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a30145d40 0x7f4a301387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2de00640 1 -- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a30145d40 msgr2=0x7f4a301387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2de00640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a30145d40 0x7f4a301387e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2de00640 1 -- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a300c58a0 msgr2=0x7f4a300c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2de00640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a300c58a0 0x7f4a300c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2de00640 1 -- 172.21.15.163:0/3143869185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a301461a0 con 0x7f4a14004090 2024-07-26T06:19:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a25400640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a30145d40 0x7f4a301387e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2e800640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a300c58a0 0x7f4a300c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2de00640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 0x7f4a30143ab0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c7a0 tx=0x7f4a2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20017070 con 0x7f4a14004090 2024-07-26T06:19:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a2000ce50 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000a8a0 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a301441b0 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.753+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a30144770 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.754+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a301f41f0 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.756+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f4a2001a070 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.757+0000 7f4a26a00640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4a0c076660 0x7f4a0c078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.757+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f4a20097060 con 0x7f4a14004090 2024-07-26T06:19:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.757+0000 7f4a25400640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4a0c076660 0x7f4a0c078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.757+0000 7f4a26a00640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7f4a0c07c520 0x7f4a0c07e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.757+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 --> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a0c07eff0 con 0x7f4a0c07c520 2024-07-26T06:19:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.757+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f4a2009e050 con 0x7f4a14004090 2024-07-26T06:19:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.758+0000 7f4a25400640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4a0c076660 0x7f4a0c078b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a180046c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.758+0000 7f4a2e800640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7f4a0c07c520 0x7f4a0c07e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.759+0000 7f4a2e800640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7f4a0c07c520 0x7f4a0c07e940 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.760+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== osd.4 v2:172.21.15.164:6826/3692835080 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a0c07eff0 con 0x7f4a0c07c520 2024-07-26T06:19:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.775+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 --> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4a300b3750 con 0x7f4a0c07c520 2024-07-26T06:19:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.776+0000 7f4a26a00640 1 -- 172.21.15.163:0/3143869185 <== osd.4 v2:172.21.15.164:6826/3692835080 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4a300b3750 con 0x7f4a0c07c520 2024-07-26T06:19:43.777 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:43.777 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:43.777 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.776+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7f4a0c07c520 msgr2=0x7f4a0c07e940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.776+0000 7f4a2f200640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.164:6826/3692835080,v1:172.21.15.164:6827/3692835080] conn(0x7f4a0c07c520 0x7f4a0c07e940 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4a0c076660 msgr2=0x7f4a0c078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f4a0c076660 0x7f4a0c078b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a180046c0 comp rx=0 tx=0).stop 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 msgr2=0x7f4a30143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 --2- 172.21.15.163:0/3143869185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a14004090 0x7f4a30143ab0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c7a0 tx=0x7f4a2000cc60 comp rx=0 tx=0).stop 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2e800640 1 -- 172.21.15.163:0/3143869185 reap_dead start 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 shutdown_connections 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 >> 172.21.15.163:0/3143869185 conn(0x7f4a300b0e30 msgr2=0x7f4a300ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 shutdown_connections 2024-07-26T06:19:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:43.777+0000 7f4a2f200640 1 -- 172.21.15.163:0/3143869185 wait complete. 2024-07-26T06:19:43.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-07-26T06:19:44.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.007+0000 7f3256600640 1 -- 172.21.15.163:0/2316194790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 msgr2=0x7f3250159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.007+0000 7f3256600640 1 --2- 172.21.15.163:0/2316194790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250159750 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f32500cb230 tx=0x7f324002a560 comp rx=0 tx=0).stop 2024-07-26T06:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.008+0000 7f3256600640 1 -- 172.21.15.163:0/2316194790 shutdown_connections 2024-07-26T06:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.008+0000 7f3256600640 1 --2- 172.21.15.163:0/2316194790 >> v2:172.21.15.164:3300/0 conn(0x7f32500cbad0 0x7f32500cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.008+0000 7f3256600640 1 --2- 172.21.15.163:0/2316194790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250159750 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.008+0000 7f3256600640 1 -- 172.21.15.163:0/2316194790 >> 172.21.15.163:0/2316194790 conn(0x7f32500b0e30 msgr2=0x7f32500b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:44.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.008+0000 7f3256600640 1 -- 172.21.15.163:0/2316194790 shutdown_connections 2024-07-26T06:19:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.008+0000 7f3256600640 1 -- 172.21.15.163:0/2316194790 wait complete. 2024-07-26T06:19:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.009+0000 7f3256600640 1 Processor -- start 2024-07-26T06:19:44.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.010+0000 7f3256600640 1 -- start start 2024-07-26T06:19:44.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.010+0000 7f3256600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32500b4420 0x7f3250158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.010+0000 7f3256600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3256600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f325014d4e0 0x7f325014d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3256600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f32500cda70 con 0x7f3250159370 2024-07-26T06:19:44.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3256600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f325006f500 con 0x7f32500b4420 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3256600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f32500cdbf0 con 0x7f325014d4e0 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34354/0 (socket says 172.21.15.163:34354) 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 -- 172.21.15.163:0/590923234 learned_addr learned my addr 172.21.15.163:0/590923234 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3255200640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32500b4420 0x7f3250158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3255c00640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f325014d4e0 0x7f325014d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 -- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f325014d4e0 msgr2=0x7f325014d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f325014d4e0 0x7f325014d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 -- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32500b4420 msgr2=0x7f3250158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32500b4420 0x7f3250158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3247e00640 1 -- 172.21.15.163:0/590923234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3250156820 con 0x7f3250159370 2024-07-26T06:19:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.011+0000 7f3255c00640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f325014d4e0 0x7f325014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:44.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.012+0000 7f3247e00640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250157290 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f323800c920 tx=0x7f323800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.012+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238007c10 con 0x7f3250159370 2024-07-26T06:19:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.012+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f323800f450 con 0x7f3250159370 2024-07-26T06:19:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.012+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323800f8a0 con 0x7f3250159370 2024-07-26T06:19:44.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.012+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32500a9670 con 0x7f3250159370 2024-07-26T06:19:44.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.012+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3250156d70 con 0x7f3250159370 2024-07-26T06:19:44.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.013+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32500cd530 con 0x7f3250159370 2024-07-26T06:19:44.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.014+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f323800fa40 con 0x7f3250159370 2024-07-26T06:19:44.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.015+0000 7f324d600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f322c076630 0x7f322c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.015+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f32380982c0 con 0x7f3250159370 2024-07-26T06:19:44.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.016+0000 7f3255200640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f322c076630 0x7f322c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.016+0000 7f3255200640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f322c076630 0x7f322c078af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3240033000 tx=0x7f3240004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.020+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3238014030 con 0x7f3250159370 2024-07-26T06:19:44.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.222+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f32501577d0 con 0x7f322c076630 2024-07-26T06:19:44.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.223+0000 7f324d600640 1 -- 172.21.15.163:0/590923234 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f32501577d0 con 0x7f322c076630 2024-07-26T06:19:44.223 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 4 to deep-scrub 2024-07-26T06:19:44.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f322c076630 msgr2=0x7f322c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f322c076630 0x7f322c078af0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3240033000 tx=0x7f3240004730 comp rx=0 tx=0).stop 2024-07-26T06:19:44.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 msgr2=0x7f3250157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250157290 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f323800c920 tx=0x7f323800cde0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 shutdown_connections 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f322c076630 0x7f322c078af0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f325014d4e0 0x7f325014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3250159370 0x7f3250157290 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 --2- 172.21.15.163:0/590923234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32500b4420 0x7f3250158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.225+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 >> 172.21.15.163:0/590923234 conn(0x7f32500b0e30 msgr2=0x7f32500b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:44.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.226+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 shutdown_connections 2024-07-26T06:19:44.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.226+0000 7f3256600640 1 -- 172.21.15.163:0/590923234 wait complete. 2024-07-26T06:19:44.245 INFO:tasks.ceph:Scrubbing osd.5 2024-07-26T06:19:44.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:44.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 -- 172.21.15.163:0/978993809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 msgr2=0x7ff64c0ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 --2- 172.21.15.163:0/978993809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c0ce2f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff640005fd0 tx=0x7ff64002a4b0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 -- 172.21.15.163:0/978993809 shutdown_connections 2024-07-26T06:19:44.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 --2- 172.21.15.163:0/978993809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff64c0ce830 0x7ff64c0b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 --2- 172.21.15.163:0/978993809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c0ce2f0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 -- 172.21.15.163:0/978993809 >> 172.21.15.163:0/978993809 conn(0x7ff64c0cfda0 msgr2=0x7ff64c0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:44.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 -- 172.21.15.163:0/978993809 shutdown_connections 2024-07-26T06:19:44.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.456+0000 7ff652600640 1 -- 172.21.15.163:0/978993809 wait complete. 2024-07-26T06:19:44.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.458+0000 7ff652600640 1 Processor -- start 2024-07-26T06:19:44.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.458+0000 7ff652600640 1 -- start start 2024-07-26T06:19:44.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff652600640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff64c0b1e80 0x7ff64c0b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff652600640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff64c0ce830 0x7ff64c1f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff652600640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c1f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff652600640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c0b5130 con 0x7ff64c15d5a0 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff652600640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c0b4fb0 con 0x7ff64c0b1e80 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff652600640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c0b52b0 con 0x7ff64c0ce830 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c1f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651c00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c1f4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34366/0 (socket says 172.21.15.163:34366) 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651c00640 1 -- 172.21.15.163:0/1976403501 learned_addr learned my addr 172.21.15.163:0/1976403501 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:44.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff64be00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff64c0ce830 0x7ff64c1f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651200640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff64c0b1e80 0x7ff64c0b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651c00640 1 -- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff64c0ce830 msgr2=0x7ff64c1f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651c00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff64c0ce830 0x7ff64c1f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.459+0000 7ff651c00640 1 -- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff64c0b1e80 msgr2=0x7ff64c0b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff651c00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff64c0b1e80 0x7ff64c0b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff651c00640 1 -- 172.21.15.163:0/1976403501 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c1f7850 con 0x7ff64c15d5a0 2024-07-26T06:19:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff651c00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c1f4240 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c7d0 tx=0x7ff64400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644003b70 con 0x7ff64c15d5a0 2024-07-26T06:19:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff644003d10 con 0x7ff64c15d5a0 2024-07-26T06:19:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64400aab0 con 0x7ff64c15d5a0 2024-07-26T06:19:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c1f7b30 con 0x7ff64c15d5a0 2024-07-26T06:19:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.460+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c1f3d20 con 0x7ff64c15d5a0 2024-07-26T06:19:44.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.461+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff64c1f3b10 con 0x7ff64c15d5a0 2024-07-26T06:19:44.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.463+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7ff64400ac50 con 0x7ff64c15d5a0 2024-07-26T06:19:44.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.463+0000 7ff648c00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff62c076630 0x7ff62c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.464+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7ff644097960 con 0x7ff64c15d5a0 2024-07-26T06:19:44.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.464+0000 7ff651200640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff62c076630 0x7ff62c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.464+0000 7ff648c00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff62c07c4f0 0x7ff62c07e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.464+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 --> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff62c07efc0 con 0x7ff62c07c4f0 2024-07-26T06:19:44.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.464+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7ff64409c2a0 con 0x7ff64c15d5a0 2024-07-26T06:19:44.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.465+0000 7ff651200640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff62c076630 0x7ff62c078af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff640033000 tx=0x7ff640004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.465+0000 7ff64be00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff62c07c4f0 0x7ff62c07e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.465+0000 7ff64be00640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff62c07c4f0 0x7ff62c07e910 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.467+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== osd.5 v2:172.21.15.164:6800/3774389112 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff62c07efc0 con 0x7ff62c07c4f0 2024-07-26T06:19:44.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.481+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 --> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff64c1f49d0 con 0x7ff62c07c4f0 2024-07-26T06:19:44.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.482+0000 7ff648c00640 1 -- 172.21.15.163:0/1976403501 <== osd.5 v2:172.21.15.164:6800/3774389112 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff64c1f49d0 con 0x7ff62c07c4f0 2024-07-26T06:19:44.484 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:44.484 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:44.485 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:44.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff62c07c4f0 msgr2=0x7ff62c07e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff652600640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.164:6800/3774389112,v1:172.21.15.164:6801/3774389112] conn(0x7ff62c07c4f0 0x7ff62c07e910 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff62c076630 msgr2=0x7ff62c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff652600640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7ff62c076630 0x7ff62c078af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff640033000 tx=0x7ff640004730 comp rx=0 tx=0).stop 2024-07-26T06:19:44.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 msgr2=0x7ff64c1f4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff652600640 1 --2- 172.21.15.163:0/1976403501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff64c15d5a0 0x7ff64c1f4240 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c7d0 tx=0x7ff64400cc90 comp rx=0 tx=0).stop 2024-07-26T06:19:44.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.483+0000 7ff651c00640 1 -- 172.21.15.163:0/1976403501 reap_dead start 2024-07-26T06:19:44.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.484+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 shutdown_connections 2024-07-26T06:19:44.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.484+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 >> 172.21.15.163:0/1976403501 conn(0x7ff64c0cfda0 msgr2=0x7ff64c15bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:44.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.484+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 shutdown_connections 2024-07-26T06:19:44.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.484+0000 7ff652600640 1 -- 172.21.15.163:0/1976403501 wait complete. 2024-07-26T06:19:44.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-07-26T06:19:44.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.702+0000 7f202e800640 1 -- 172.21.15.163:0/2760739272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f202815d5a0 msgr2=0x7f20280ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.702+0000 7f202e800640 1 --2- 172.21.15.163:0/2760739272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f202815d5a0 0x7f20280ce2f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2028160e50 tx=0x7f201c02a560 comp rx=0 tx=0).stop 2024-07-26T06:19:44.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.702+0000 7f202e800640 1 -- 172.21.15.163:0/2760739272 shutdown_connections 2024-07-26T06:19:44.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.702+0000 7f202e800640 1 --2- 172.21.15.163:0/2760739272 >> v2:172.21.15.164:3300/0 conn(0x7f20280ce830 0x7f20280b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.702+0000 7f202e800640 1 --2- 172.21.15.163:0/2760739272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f202815d5a0 0x7f20280ce2f0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.702+0000 7f202e800640 1 -- 172.21.15.163:0/2760739272 >> 172.21.15.163:0/2760739272 conn(0x7f20280cfda0 msgr2=0x7f20280d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:44.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.703+0000 7f202e800640 1 -- 172.21.15.163:0/2760739272 shutdown_connections 2024-07-26T06:19:44.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.703+0000 7f202e800640 1 -- 172.21.15.163:0/2760739272 wait complete. 2024-07-26T06:19:44.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.704+0000 7f202e800640 1 Processor -- start 2024-07-26T06:19:44.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.704+0000 7f202e800640 1 -- start start 2024-07-26T06:19:44.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.705+0000 7f202e800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20280b1e80 0x7f2028159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.705+0000 7f202e800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 0x7f20280c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.705+0000 7f202e800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f202815d5a0 0x7f20280cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.705+0000 7f202e800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20280b5130 con 0x7f20280ce830 2024-07-26T06:19:44.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.705+0000 7f202e800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20280b4fb0 con 0x7f20280b1e80 2024-07-26T06:19:44.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.705+0000 7f202e800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20280b52b0 con 0x7f202815d5a0 2024-07-26T06:19:44.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 0x7f20280c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 0x7f20280c7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:34378/0 (socket says 172.21.15.163:34378) 2024-07-26T06:19:44.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 -- 172.21.15.163:0/3874882841 learned_addr learned my addr 172.21.15.163:0/3874882841 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:44.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202d400640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20280b1e80 0x7f2028159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202de00640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f202815d5a0 0x7f20280cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 -- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f202815d5a0 msgr2=0x7f20280cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f202815d5a0 0x7f20280cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 -- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20280b1e80 msgr2=0x7f2028159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20280b1e80 0x7f2028159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 -- 172.21.15.163:0/3874882841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20280c8770 con 0x7f20280ce830 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.706+0000 7f202ca00640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 0x7f20280c7830 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f201400b4a0 tx=0x7f201400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.707+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014014070 con 0x7f20280ce830 2024-07-26T06:19:44.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.707+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2014010700 con 0x7f20280ce830 2024-07-26T06:19:44.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.707+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20280c81b0 con 0x7f20280ce830 2024-07-26T06:19:44.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.707+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014010b50 con 0x7f20280ce830 2024-07-26T06:19:44.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.707+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2028147650 con 0x7f20280ce830 2024-07-26T06:19:44.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.708+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20280ce2f0 con 0x7f20280ce830 2024-07-26T06:19:44.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.709+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f2014017070 con 0x7f20280ce830 2024-07-26T06:19:44.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.710+0000 7f2025600640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2008076630 0x7f2008078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:44.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.710+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f20140975c0 con 0x7f20280ce830 2024-07-26T06:19:44.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.710+0000 7f202d400640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2008076630 0x7f2008078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:44.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.711+0000 7f202d400640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2008076630 0x7f2008078af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f201c033000 tx=0x7f201c004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:44.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.714+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2014060cd0 con 0x7f20280ce830 2024-07-26T06:19:44.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.920+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2028159ed0 con 0x7f2008076630 2024-07-26T06:19:44.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.921+0000 7f2025600640 1 -- 172.21.15.163:0/3874882841 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2028159ed0 con 0x7f2008076630 2024-07-26T06:19:44.922 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 5 to deep-scrub 2024-07-26T06:19:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.923+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2008076630 msgr2=0x7f2008078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.923+0000 7f202e800640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2008076630 0x7f2008078af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f201c033000 tx=0x7f201c004730 comp rx=0 tx=0).stop 2024-07-26T06:19:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.923+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 msgr2=0x7f20280c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.923+0000 7f202e800640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 0x7f20280c7830 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f201400b4a0 tx=0x7f201400b960 comp rx=0 tx=0).stop 2024-07-26T06:19:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 shutdown_connections 2024-07-26T06:19:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f2008076630 0x7f2008078af0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f202815d5a0 0x7f20280cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20280ce830 0x7f20280c7830 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 --2- 172.21.15.163:0/3874882841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20280b1e80 0x7f2028159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 >> 172.21.15.163:0/3874882841 conn(0x7f20280cfda0 msgr2=0x7f20280b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 shutdown_connections 2024-07-26T06:19:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:44.924+0000 7f202e800640 1 -- 172.21.15.163:0/3874882841 wait complete. 2024-07-26T06:19:44.943 INFO:tasks.ceph:Scrubbing osd.6 2024-07-26T06:19:44.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:45.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16cae00640 1 -- 172.21.15.163:0/1178555599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 msgr2=0x7f16ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16cae00640 1 --2- 172.21.15.163:0/1178555599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 0x7f16ac004800 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f16b0006fa0 tx=0x7f16ac005830 comp rx=0 tx=0).stop 2024-07-26T06:19:45.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16ca400640 1 -- 172.21.15.163:0/1178555599 reap_dead start 2024-07-26T06:19:45.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16cae00640 1 -- 172.21.15.163:0/1178555599 shutdown_connections 2024-07-26T06:19:45.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16cae00640 1 -- 172.21.15.163:0/1178555599 >> 172.21.15.163:0/1178555599 conn(0x7f16c40b0e30 msgr2=0x7f16c40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:45.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16cae00640 1 -- 172.21.15.163:0/1178555599 shutdown_connections 2024-07-26T06:19:45.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.151+0000 7f16cae00640 1 -- 172.21.15.163:0/1178555599 wait complete. 2024-07-26T06:19:45.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.153+0000 7f16cae00640 1 Processor -- start 2024-07-26T06:19:45.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.153+0000 7f16cae00640 1 -- start start 2024-07-26T06:19:45.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.153+0000 7f16cae00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 0x7f16c4163a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16cae00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c4161f40 0x7f16c41f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16cae00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16c41f6ca0 0x7f16c412e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16cae00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4166310 con 0x7f16ac004100 2024-07-26T06:19:45.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16cae00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4166190 con 0x7f16c4161f40 2024-07-26T06:19:45.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16cae00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f16c4166490 con 0x7f16c41f6ca0 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 0x7f16c4163a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 0x7f16c4163a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:60660/0 (socket says 172.21.15.163:60660) 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 -- 172.21.15.163:0/2902431655 learned_addr learned my addr 172.21.15.163:0/2902431655 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9000640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c4161f40 0x7f16c41f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16ca400640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16c41f6ca0 0x7f16c412e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 -- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16c41f6ca0 msgr2=0x7f16c412e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16c41f6ca0 0x7f16c412e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 -- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c4161f40 msgr2=0x7f16c41f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c4161f40 0x7f16c41f6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9a00640 1 -- 172.21.15.163:0/2902431655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c41623c0 con 0x7f16ac004100 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.154+0000 7f16c9000640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c4161f40 0x7f16c41f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.155+0000 7f16c9a00640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 0x7f16c4163a30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f16b800c530 tx=0x7f16b800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.155+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b80052d0 con 0x7f16ac004100 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.155+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16b8005470 con 0x7f16ac004100 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.155+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c412c4d0 con 0x7f16ac004100 2024-07-26T06:19:45.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.155+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8012850 con 0x7f16ac004100 2024-07-26T06:19:45.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.155+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c412ca90 con 0x7f16ac004100 2024-07-26T06:19:45.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.156+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16c412c280 con 0x7f16ac004100 2024-07-26T06:19:45.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.157+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f16b80129f0 con 0x7f16ac004100 2024-07-26T06:19:45.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.158+0000 7f16c1600640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f16a4076630 0x7f16a4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.158+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f16b80976d0 con 0x7f16ac004100 2024-07-26T06:19:45.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.159+0000 7f16c1600640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7f16a407c4f0 0x7f16a407e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.159+0000 7f16c9000640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f16a4076630 0x7f16a4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.159+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 --> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16a407efc0 con 0x7f16a407c4f0 2024-07-26T06:19:45.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.159+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f16b809e050 con 0x7f16ac004100 2024-07-26T06:19:45.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.159+0000 7f16ca400640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7f16a407c4f0 0x7f16a407e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.159+0000 7f16c9000640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f16a4076630 0x7f16a4078af0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f16b0033000 tx=0x7f16b00019c0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.160+0000 7f16ca400640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7f16a407c4f0 0x7f16a407e910 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.162+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== osd.6 v2:172.21.15.164:6808/2876752734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a407efc0 con 0x7f16a407c4f0 2024-07-26T06:19:45.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.177+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 --> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f16c40cbd40 con 0x7f16a407c4f0 2024-07-26T06:19:45.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16c1600640 1 -- 172.21.15.163:0/2902431655 <== osd.6 v2:172.21.15.164:6808/2876752734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f16c40cbd40 con 0x7f16a407c4f0 2024-07-26T06:19:45.180 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:45.180 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:45.180 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:45.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7f16a407c4f0 msgr2=0x7f16a407e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16cae00640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.164:6808/2876752734,v1:172.21.15.164:6809/2876752734] conn(0x7f16a407c4f0 0x7f16a407e910 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f16a4076630 msgr2=0x7f16a4078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16cae00640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f16a4076630 0x7f16a4078af0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f16b0033000 tx=0x7f16b00019c0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 msgr2=0x7f16c4163a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16cae00640 1 --2- 172.21.15.163:0/2902431655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16ac004100 0x7f16c4163a30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f16b800c530 tx=0x7f16b800c9f0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.179+0000 7f16ca400640 1 -- 172.21.15.163:0/2902431655 reap_dead start 2024-07-26T06:19:45.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.180+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 shutdown_connections 2024-07-26T06:19:45.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.180+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 >> 172.21.15.163:0/2902431655 conn(0x7f16c40b0e30 msgr2=0x7f16c40ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:45.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.180+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 shutdown_connections 2024-07-26T06:19:45.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.180+0000 7f16cae00640 1 -- 172.21.15.163:0/2902431655 wait complete. 2024-07-26T06:19:45.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-07-26T06:19:45.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.404+0000 7fddabe00640 1 -- 172.21.15.163:0/2645341296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 msgr2=0x7fdd8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.404+0000 7fddabe00640 1 --2- 172.21.15.163:0/2645341296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdd8c00b9c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c005fd0 tx=0x7fdd8c005830 comp rx=0 tx=0).stop 2024-07-26T06:19:45.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.404+0000 7fdda9e00640 1 -- 172.21.15.163:0/2645341296 reap_dead start 2024-07-26T06:19:45.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.404+0000 7fddabe00640 1 -- 172.21.15.163:0/2645341296 shutdown_connections 2024-07-26T06:19:45.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.404+0000 7fddabe00640 1 -- 172.21.15.163:0/2645341296 >> 172.21.15.163:0/2645341296 conn(0x7fdda409c7e0 msgr2=0x7fdda408ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:45.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.404+0000 7fddabe00640 1 -- 172.21.15.163:0/2645341296 shutdown_connections 2024-07-26T06:19:45.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.405+0000 7fddabe00640 1 -- 172.21.15.163:0/2645341296 wait complete. 2024-07-26T06:19:45.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.406+0000 7fddabe00640 1 Processor -- start 2024-07-26T06:19:45.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.406+0000 7fddabe00640 1 -- start start 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fddabe00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdda415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fddabe00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdda41e8570 0x7fdda41e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fddabe00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdda415b240 0x7fdda415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fddabe00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdda40d3570 con 0x7fdd8c0051c0 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fddabe00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdda40d33f0 con 0x7fdda415b240 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fddabe00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdda40b4070 con 0x7fdda41e8570 2024-07-26T06:19:45.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.407+0000 7fdda9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdda415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdda415deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:60678/0 (socket says 172.21.15.163:60678) 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 -- 172.21.15.163:0/2079297229 learned_addr learned my addr 172.21.15.163:0/2079297229 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9e00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdda415b240 0x7fdda415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 -- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdda41e8570 msgr2=0x7fdda41e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdda41e8570 0x7fdda41e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 -- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdda415b240 msgr2=0x7fdda415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdda415b240 0x7fdda415b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 -- 172.21.15.163:0/2079297229 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda40b72e0 con 0x7fdd8c0051c0 2024-07-26T06:19:45.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda9400640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdda415deb0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800d7b0 tx=0x7fdd9800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.408+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98018070 con 0x7fdd8c0051c0 2024-07-26T06:19:45.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.409+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd980052e0 con 0x7fdd8c0051c0 2024-07-26T06:19:45.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.409+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9800b920 con 0x7fdd8c0051c0 2024-07-26T06:19:45.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.409+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda415fcc0 con 0x7fdd8c0051c0 2024-07-26T06:19:45.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.409+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda4160240 con 0x7fdd8c0051c0 2024-07-26T06:19:45.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.410+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda40ceb50 con 0x7fdd8c0051c0 2024-07-26T06:19:45.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.411+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fdd9801b070 con 0x7fdd8c0051c0 2024-07-26T06:19:45.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.412+0000 7fdda1600640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdd84076630 0x7fdd84078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.412+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fdd98097330 con 0x7fdd8c0051c0 2024-07-26T06:19:45.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.412+0000 7fdda8a00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdd84076630 0x7fdd84078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.413+0000 7fdda8a00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdd84076630 0x7fdd84078af0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdda40cda20 tx=0x7fdd9000d3f0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.416+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdda40b72e0 con 0x7fdd8c0051c0 2024-07-26T06:19:45.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.623+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fdda415bc00 con 0x7fdd84076630 2024-07-26T06:19:45.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.624+0000 7fdda1600640 1 -- 172.21.15.163:0/2079297229 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdda415bc00 con 0x7fdd84076630 2024-07-26T06:19:45.625 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 6 to deep-scrub 2024-07-26T06:19:45.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdd84076630 msgr2=0x7fdd84078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdd84076630 0x7fdd84078af0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdda40cda20 tx=0x7fdd9000d3f0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 msgr2=0x7fdda415deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdda415deb0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800d7b0 tx=0x7fdd9800dc70 comp rx=0 tx=0).stop 2024-07-26T06:19:45.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 shutdown_connections 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdd84076630 0x7fdd84078af0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdda415b240 0x7fdda415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdda41e8570 0x7fdda41e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 --2- 172.21.15.163:0/2079297229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd8c0051c0 0x7fdda415deb0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 >> 172.21.15.163:0/2079297229 conn(0x7fdda409c7e0 msgr2=0x7fdda40c8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 shutdown_connections 2024-07-26T06:19:45.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.627+0000 7fddabe00640 1 -- 172.21.15.163:0/2079297229 wait complete. 2024-07-26T06:19:45.648 INFO:tasks.ceph:Scrubbing osd.7 2024-07-26T06:19:45.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-07-26T06:19:45.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.854+0000 7f0903400640 1 -- 172.21.15.163:0/1367943264 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f80047a0 con 0x7f0904159370 2024-07-26T06:19:45.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.854+0000 7f090a400640 1 -- 172.21.15.163:0/1367943264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 msgr2=0x7f0904159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.854+0000 7f090a400640 1 --2- 172.21.15.163:0/1367943264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f0904159750 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f08f8005fd0 tx=0x7f08f802a530 comp rx=0 tx=0).stop 2024-07-26T06:19:45.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.855+0000 7f090a400640 1 -- 172.21.15.163:0/1367943264 shutdown_connections 2024-07-26T06:19:45.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.855+0000 7f090a400640 1 --2- 172.21.15.163:0/1367943264 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09040b4420 0x7f09040b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.855+0000 7f090a400640 1 --2- 172.21.15.163:0/1367943264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f0904159750 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.855+0000 7f090a400640 1 -- 172.21.15.163:0/1367943264 >> 172.21.15.163:0/1367943264 conn(0x7f09040b0e30 msgr2=0x7f09040b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:45.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.855+0000 7f090a400640 1 -- 172.21.15.163:0/1367943264 shutdown_connections 2024-07-26T06:19:45.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.855+0000 7f090a400640 1 -- 172.21.15.163:0/1367943264 wait complete. 2024-07-26T06:19:45.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.856+0000 7f090a400640 1 Processor -- start 2024-07-26T06:19:45.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.857+0000 7f090a400640 1 -- start start 2024-07-26T06:19:45.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.857+0000 7f090a400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09040b4420 0x7f090415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.857+0000 7f090a400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f09040ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f090a400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f090415b5e0 0x7f09040ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f090a400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f090406f2f0 con 0x7f0904159370 2024-07-26T06:19:45.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f090a400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f09040cd210 con 0x7f090415b5e0 2024-07-26T06:19:45.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f090a400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f090406f470 con 0x7f09040b4420 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f09040ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f09040ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:60694/0 (socket says 172.21.15.163:60694) 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 -- 172.21.15.163:0/1357920929 learned_addr learned my addr 172.21.15.163:0/1357920929 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0909000640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09040b4420 0x7f090415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0909a00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f090415b5e0 0x7f09040ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 -- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09040b4420 msgr2=0x7f090415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09040b4420 0x7f090415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 -- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f090415b5e0 msgr2=0x7f09040ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f090415b5e0 0x7f09040ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0903e00640 1 -- 172.21.15.163:0/1357920929 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090415bab0 con 0x7f0904159370 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.858+0000 7f0909a00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f090415b5e0 0x7f09040ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.859+0000 7f0903e00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f09040ad8e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c410 tx=0x7f08f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.859+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0017070 con 0x7f0904159370 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.859+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08f00035f0 con 0x7f0904159370 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.859+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090415bd90 con 0x7f0904159370 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.859+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0003a80 con 0x7f0904159370 2024-07-26T06:19:45.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.859+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090415c350 con 0x7f0904159370 2024-07-26T06:19:45.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.860+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09040b2130 con 0x7f0904159370 2024-07-26T06:19:45.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.861+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f08f001a070 con 0x7f0904159370 2024-07-26T06:19:45.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.862+0000 7f0900c00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08e4076630 0x7f08e4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.862+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f08f0096df0 con 0x7f0904159370 2024-07-26T06:19:45.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.862+0000 7f0909000640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08e4076630 0x7f08e4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.863+0000 7f0900c00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f08e407c4f0 0x7f08e407e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:45.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.863+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 --> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08e407efc0 con 0x7f08e407c4f0 2024-07-26T06:19:45.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.863+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=104) ==== 24+0+0 (secure 0 0 0) 0x7f08f009c400 con 0x7f0904159370 2024-07-26T06:19:45.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.863+0000 7f0909000640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08e4076630 0x7f08e4078af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08f8033000 tx=0x7f08f8004730 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.863+0000 7f0909a00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f08e407c4f0 0x7f08e407e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:45.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.864+0000 7f0909a00640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f08e407c4f0 0x7f08e407e910 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:45.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.866+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== osd.7 v2:172.21.15.164:6818/734207166 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08e407efc0 con 0x7f08e407c4f0 2024-07-26T06:19:45.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.881+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 --> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0904074210 con 0x7f08e407c4f0 2024-07-26T06:19:45.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f0900c00640 1 -- 172.21.15.163:0/1357920929 <== osd.7 v2:172.21.15.164:6818/734207166 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0904074210 con 0x7f08e407c4f0 2024-07-26T06:19:45.883 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:19:45.884 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-07-26T06:19:45.884 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:19:45.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f08e407c4f0 msgr2=0x7f08e407e910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f090a400640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.164:6818/734207166,v1:172.21.15.164:6819/734207166] conn(0x7f08e407c4f0 0x7f08e407e910 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08e4076630 msgr2=0x7f08e4078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f090a400640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f08e4076630 0x7f08e4078af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08f8033000 tx=0x7f08f8004730 comp rx=0 tx=0).stop 2024-07-26T06:19:45.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 msgr2=0x7f09040ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:45.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f090a400640 1 --2- 172.21.15.163:0/1357920929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0904159370 0x7f09040ad8e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c410 tx=0x7f08f000c8d0 comp rx=0 tx=0).stop 2024-07-26T06:19:45.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.883+0000 7f0909a00640 1 -- 172.21.15.163:0/1357920929 reap_dead start 2024-07-26T06:19:45.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.884+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 shutdown_connections 2024-07-26T06:19:45.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.884+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 >> 172.21.15.163:0/1357920929 conn(0x7f09040b0e30 msgr2=0x7f09040b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:45.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.884+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 shutdown_connections 2024-07-26T06:19:45.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:45.884+0000 7f090a400640 1 -- 172.21.15.163:0/1357920929 wait complete. 2024-07-26T06:19:45.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-07-26T06:19:46.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 -- 172.21.15.163:0/348788451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 msgr2=0x7f63900b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 --2- 172.21.15.163:0/348788451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 0x7f63900b6c00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6388005fd0 tx=0x7f638802a660 comp rx=0 tx=0).stop 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 -- 172.21.15.163:0/348788451 shutdown_connections 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 --2- 172.21.15.163:0/348788451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 0x7f63900b6c00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 --2- 172.21.15.163:0/348788451 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63900cfe30 0x7f63900d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 -- 172.21.15.163:0/348788451 >> 172.21.15.163:0/348788451 conn(0x7f63900b4fa0 msgr2=0x7f63900b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 -- 172.21.15.163:0/348788451 shutdown_connections 2024-07-26T06:19:46.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.101+0000 7f6396800640 1 -- 172.21.15.163:0/348788451 wait complete. 2024-07-26T06:19:46.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.103+0000 7f6396800640 1 Processor -- start 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.103+0000 7f6396800640 1 -- start start 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f6396800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 0x7f639006c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f6396800640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 0x7f639015a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f6396800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f639015d540 0x7f639015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f6396800640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f639015daa0 con 0x7f63900cfe30 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f6396800640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f639015d920 con 0x7f63900b6790 2024-07-26T06:19:46.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f6396800640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63900b7140 con 0x7f639015d540 2024-07-26T06:19:46.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f638ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 0x7f639015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f638ea00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 0x7f639015a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:60710/0 (socket says 172.21.15.163:60710) 2024-07-26T06:19:46.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f638ea00640 1 -- 172.21.15.163:0/2877001756 learned_addr learned my addr 172.21.15.163:0/2877001756 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:46.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f638fe00640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f639015d540 0x7f639015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.104+0000 7f638f400640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 0x7f639006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638ea00640 1 -- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f639015d540 msgr2=0x7f639015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638ea00640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f639015d540 0x7f639015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638ea00640 1 -- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 msgr2=0x7f639006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638ea00640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 0x7f639006c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638ea00640 1 -- 172.21.15.163:0/2877001756 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639015b0f0 con 0x7f63900cfe30 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638fe00640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f639015d540 0x7f639015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f638ea00640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 0x7f639015a580 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f637800ea80 tx=0x7f637800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637800ce40 con 0x7f63900cfe30 2024-07-26T06:19:46.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6378011560 con 0x7f63900cfe30 2024-07-26T06:19:46.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639015eb60 con 0x7f63900cfe30 2024-07-26T06:19:46.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6378011950 con 0x7f63900cfe30 2024-07-26T06:19:46.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.105+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639015e080 con 0x7f63900cfe30 2024-07-26T06:19:46.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.106+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63900b6c00 con 0x7f63900cfe30 2024-07-26T06:19:46.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.108+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f6378011af0 con 0x7f63900cfe30 2024-07-26T06:19:46.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.109+0000 7f6387400640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f636c076630 0x7f636c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.109+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f6378098940 con 0x7f63900cfe30 2024-07-26T06:19:46.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.109+0000 7f638f400640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f636c076630 0x7f636c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.110+0000 7f638f400640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f636c076630 0x7f636c078af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63900cccf0 tx=0x7f6380009040 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:46.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.114+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6378016030 con 0x7f63900cfe30 2024-07-26T06:19:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.321+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f63900c6d20 con 0x7f636c076630 2024-07-26T06:19:46.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.322+0000 7f6387400640 1 -- 172.21.15.163:0/2877001756 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f63900c6d20 con 0x7f636c076630 2024-07-26T06:19:46.323 INFO:teuthology.orchestra.run.smithi163.stderr:instructed osd(s) 7 to deep-scrub 2024-07-26T06:19:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f636c076630 msgr2=0x7f636c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f636c076630 0x7f636c078af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63900cccf0 tx=0x7f6380009040 comp rx=0 tx=0).stop 2024-07-26T06:19:46.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 msgr2=0x7f639015a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 0x7f639015a580 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f637800ea80 tx=0x7f637800ef40 comp rx=0 tx=0).stop 2024-07-26T06:19:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 shutdown_connections 2024-07-26T06:19:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f636c076630 0x7f636c078af0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f639015d540 0x7f639015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63900cfe30 0x7f639015a580 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 --2- 172.21.15.163:0/2877001756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63900b6790 0x7f639006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 >> 172.21.15.163:0/2877001756 conn(0x7f63900b4fa0 msgr2=0x7f63900ca520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:46.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 shutdown_connections 2024-07-26T06:19:46.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.325+0000 7f6396800640 1 -- 172.21.15.163:0/2877001756 wait complete. 2024-07-26T06:19:46.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75a00640 1 -- 172.21.15.163:0/2451577471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e54004d40 msgr2=0x7f0e540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75a00640 1 --2- 172.21.15.163:0/2451577471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e54004d40 0x7f0e540051c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0e60001050 tx=0x7f0e54005830 comp rx=0 tx=0).stop 2024-07-26T06:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75000640 1 -- 172.21.15.163:0/2451577471 reap_dead start 2024-07-26T06:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75a00640 1 -- 172.21.15.163:0/2451577471 shutdown_connections 2024-07-26T06:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75a00640 1 -- 172.21.15.163:0/2451577471 >> 172.21.15.163:0/2451577471 conn(0x7f0e700b0e30 msgr2=0x7f0e700b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75a00640 1 -- 172.21.15.163:0/2451577471 shutdown_connections 2024-07-26T06:19:46.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.545+0000 7f0e75a00640 1 -- 172.21.15.163:0/2451577471 wait complete. 2024-07-26T06:19:46.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.546+0000 7f0e75a00640 1 Processor -- start 2024-07-26T06:19:46.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.547+0000 7f0e75a00640 1 -- start start 2024-07-26T06:19:46.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.547+0000 7f0e75a00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e54004d40 0x7f0e701e4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.547+0000 7f0e75a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 0x7f0e7015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.547+0000 7f0e75a00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e7015a590 0x7f0e7014d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.547+0000 7f0e75a00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0e700b3e20 con 0x7f0e701e9490 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e75a00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0e700cdfe0 con 0x7f0e54004d40 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e75a00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0e700b3fa0 con 0x7f0e7015a590 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 0x7f0e7015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 0x7f0e7015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:60724/0 (socket says 172.21.15.163:60724) 2024-07-26T06:19:46.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 -- 172.21.15.163:0/990092888 learned_addr learned my addr 172.21.15.163:0/990092888 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e75000640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e7015a590 0x7f0e7014d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6fe00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e54004d40 0x7f0e701e4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 -- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e7015a590 msgr2=0x7f0e7014d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e7015a590 0x7f0e7014d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 -- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e54004d40 msgr2=0x7f0e701e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e54004d40 0x7f0e701e4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6f400640 1 -- 172.21.15.163:0/990092888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7014ec80 con 0x7f0e701e9490 2024-07-26T06:19:46.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.548+0000 7f0e6fe00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e54004d40 0x7f0e701e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:19:46.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.549+0000 7f0e6f400640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 0x7f0e7015a050 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c500 tx=0x7f0e5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:46.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.549+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c016020 con 0x7f0e701e9490 2024-07-26T06:19:46.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.549+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e5c0053e0 con 0x7f0e701e9490 2024-07-26T06:19:46.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.549+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c007280 con 0x7f0e701e9490 2024-07-26T06:19:46.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.549+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7014dff0 con 0x7f0e701e9490 2024-07-26T06:19:46.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.549+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7014e5b0 con 0x7f0e701e9490 2024-07-26T06:19:46.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.550+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e700b4500 con 0x7f0e701e9490 2024-07-26T06:19:46.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.551+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f0e5c007420 con 0x7f0e701e9490 2024-07-26T06:19:46.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.552+0000 7f0e67e00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e50076660 0x7f0e50078b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:19:46.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.552+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f0e5c096f80 con 0x7f0e701e9490 2024-07-26T06:19:46.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.553+0000 7f0e6fe00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e50076660 0x7f0e50078b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:19:46.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.553+0000 7f0e6fe00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e50076660 0x7f0e50078b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0e68005fd0 tx=0x7f0e68009500 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:19:46.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.557+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e5c060860 con 0x7f0e701e9490 2024-07-26T06:19:46.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.762+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e700b34f0 con 0x7f0e50076660 2024-07-26T06:19:46.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.764+0000 7f0e67e00640 1 -- 172.21.15.163:0/990092888 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13362 (secure 0 0 0) 0x7f0e700b34f0 con 0x7f0e50076660 2024-07-26T06:19:46.766 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:19:46.766 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:19:46.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e50076660 msgr2=0x7f0e50078b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e50076660 0x7f0e50078b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0e68005fd0 tx=0x7f0e68009500 comp rx=0 tx=0).stop 2024-07-26T06:19:46.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 msgr2=0x7f0e7015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:19:46.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 0x7f0e7015a050 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c500 tx=0x7f0e5c00c9c0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 shutdown_connections 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f0e50076660 0x7f0e50078b20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e7015a590 0x7f0e7014d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0e701e9490 0x7f0e7015a050 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 --2- 172.21.15.163:0/990092888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e54004d40 0x7f0e701e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 >> 172.21.15.163:0/990092888 conn(0x7f0e700b0e30 msgr2=0x7f0e7015c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 shutdown_connections 2024-07-26T06:19:46.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:19:46.767+0000 7f0e75a00640 1 -- 172.21.15.163:0/990092888 wait complete. 2024-07-26T06:19:46.786 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-07-26T06:19:46.488056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260732,"kb_used_data":11272,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713988,"statfs":{"total":773094113280,"available":772827123712,"internally_reserved":0,"allocated":11542528,"data_stored":5697104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003003"},"pg_stats":[{"pgid":"1.0","version":"70'79","reported_seq":291,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-07-26T06:19:30.862750+0000","last_change":"2024-07-26T06:13:23.455536+0000","last_active":"2024-07-26T06:19:30.862750+0000","last_peered":"2024-07-26T06:19:30.862750+0000","last_clean":"2024-07-26T06:19:30.862750+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:19:30.862750+0000","last_undegraded":"2024-07-26T06:19:30.862750+0000","last_fullsized":"2024-07-26T06:19:30.862750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:13:21.818251+0000","last_clean_scrub_stamp":"2024-07-26T06:13:21.818251+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:52:11.978896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509520,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509521,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509522,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509521,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:19:46.789 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-07-26T06:13:21.818251+0000 time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=6, tm_min=13, tm_sec=21, tm_wday=4, tm_yday=208, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=7, tm_mday=26, tm_hour=6, tm_min=19, tm_sec=39, tm_wday=4, tm_yday=208, tm_isdst=0) 2024-07-26T06:19:46.789 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-07-26T06:19:57.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:19:57.433 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:20:06.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T06:20:07.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 -- 172.21.15.163:0/4283472203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40cbad0 msgr2=0x7f7fb40cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/4283472203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40cbad0 0x7f7fb40cd530 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c005fd0 tx=0x7f7f9c02a530 comp rx=0 tx=0).stop 2024-07-26T06:20:07.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 -- 172.21.15.163:0/4283472203 shutdown_connections 2024-07-26T06:20:07.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/4283472203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40cbad0 0x7f7fb40cd530 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/4283472203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fb4159370 0x7f7fb4159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 -- 172.21.15.163:0/4283472203 >> 172.21.15.163:0/4283472203 conn(0x7f7fb40b0e30 msgr2=0x7f7fb40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:20:07.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 -- 172.21.15.163:0/4283472203 shutdown_connections 2024-07-26T06:20:07.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.013+0000 7f7fb3e00640 1 -- 172.21.15.163:0/4283472203 wait complete. 2024-07-26T06:20:07.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.015+0000 7f7fb3e00640 1 Processor -- start 2024-07-26T06:20:07.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.015+0000 7f7fb3e00640 1 -- start start 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3e00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fb40b4420 0x7f7fb4154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3e00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fb4159370 0x7f7fb415c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 0x7f7fb415a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3e00640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb40cda70 con 0x7f7fb40adb50 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3e00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb406f500 con 0x7f7fb40b4420 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3e00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb40cdbf0 con 0x7f7fb4159370 2024-07-26T06:20:07.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 0x7f7fb415a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 0x7f7fb415a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52610/0 (socket says 172.21.15.163:52610) 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.016+0000 7f7fb3400640 1 -- 172.21.15.163:0/1561658580 learned_addr learned my addr 172.21.15.163:0/1561658580 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fb3400640 1 -- 172.21.15.163:0/1561658580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fb4159370 msgr2=0x7f7fb415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fb3400640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fb4159370 0x7f7fb415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fb3400640 1 -- 172.21.15.163:0/1561658580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fb40b4420 msgr2=0x7f7fb4154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fb3400640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fb40b4420 0x7f7fb4154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fb3400640 1 -- 172.21.15.163:0/1561658580 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb414c5b0 con 0x7f7fb40adb50 2024-07-26T06:20:07.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fb3400640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 0x7f7fb415a990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00c730 tx=0x7f7fac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:20:07.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac016020 con 0x7f7fb40adb50 2024-07-26T06:20:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.017+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fac00ab40 con 0x7f7fb40adb50 2024-07-26T06:20:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.018+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00ce40 con 0x7f7fb40adb50 2024-07-26T06:20:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.018+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb414de30 con 0x7f7fb40adb50 2024-07-26T06:20:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.018+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb415c890 con 0x7f7fb40adb50 2024-07-26T06:20:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.019+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb40cd530 con 0x7f7fb40adb50 2024-07-26T06:20:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.020+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f7fac019020 con 0x7f7fb40adb50 2024-07-26T06:20:07.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.021+0000 7f7fab400640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7f90076630 0x7f7f90078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.021+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f7fac097da0 con 0x7f7fb40adb50 2024-07-26T06:20:07.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.021+0000 7f7fb2a00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7f90076630 0x7f7f90078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.022+0000 7f7fb2a00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7f90076630 0x7f7f90078af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400a8f0 tx=0x7f7fa4005bd0 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:20:07.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.025+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7fac020020 con 0x7f7fb40adb50 2024-07-26T06:20:07.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.228+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 --> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7fb414c7b0 con 0x7f7f90076630 2024-07-26T06:20:07.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.229+0000 7f7fab400640 1 -- 172.21.15.163:0/1561658580 <== mgr.4118 v2:172.21.15.163:6836/3263392688 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13369 (secure 0 0 0) 0x7f7fa400a500 con 0x7f7f90076630 2024-07-26T06:20:07.230 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:20:07.230 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.231+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7f90076630 msgr2=0x7f7f90078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.231+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7f90076630 0x7f7f90078af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400a8f0 tx=0x7f7fa4005bd0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 msgr2=0x7f7fb415a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 0x7f7fb415a990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00c730 tx=0x7f7fac00cbf0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 shutdown_connections 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f7f90076630 0x7f7f90078af0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7fb40adb50 0x7f7fb415a990 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fb4159370 0x7f7fb415c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 --2- 172.21.15.163:0/1561658580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fb40b4420 0x7f7fb4154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 >> 172.21.15.163:0/1561658580 conn(0x7f7fb40b0e30 msgr2=0x7f7fb40b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:20:07.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 shutdown_connections 2024-07-26T06:20:07.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.232+0000 7f7fb3e00640 1 -- 172.21.15.163:0/1561658580 wait complete. 2024-07-26T06:20:07.252 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-07-26T06:20:06.494220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260720,"kb_used_data":11272,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714000,"statfs":{"total":773094113280,"available":772827136000,"internally_reserved":0,"allocated":11542528,"data_stored":5697104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003546"},"pg_stats":[{"pgid":"1.0","version":"104'81","reported_seq":302,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-07-26T06:19:46.847819+0000","last_change":"2024-07-26T06:19:46.847894+0000","last_active":"2024-07-26T06:19:46.847819+0000","last_peered":"2024-07-26T06:19:46.847819+0000","last_clean":"2024-07-26T06:19:46.847819+0000","last_became_active":"2024-07-26T06:13:23.452508+0000","last_became_peered":"2024-07-26T06:13:23.452508+0000","last_unstale":"2024-07-26T06:19:46.847819+0000","last_undegraded":"2024-07-26T06:19:46.847819+0000","last_fullsized":"2024-07-26T06:19:46.847819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'81","last_scrub_stamp":"2024-07-26T06:19:46.847628+0000","last_deep_scrub":"104'81","last_deep_scrub_stamp":"2024-07-26T06:19:46.847628+0000","last_clean_scrub_stamp":"2024-07-26T06:19:46.847628+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:15.487801+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509524,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33020,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509525,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542228,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542227,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509526,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":564546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33020,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1888256,"data_stored":1154914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:20:07.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-07-26T06:20:07.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 -- 172.21.15.163:0/2540672596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 msgr2=0x7f9fec0b6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2540672596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0b6520 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0b6060 tx=0x7f9fe402a5d0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 -- 172.21.15.163:0/2540672596 shutdown_connections 2024-07-26T06:20:07.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2540672596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0b6520 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2540672596 >> v2:172.21.15.164:3300/0 conn(0x7f9fec0d3030 0x7f9fec0cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 -- 172.21.15.163:0/2540672596 >> 172.21.15.163:0/2540672596 conn(0x7f9fec0b4fe0 msgr2=0x7f9fec0b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:20:07.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 -- 172.21.15.163:0/2540672596 shutdown_connections 2024-07-26T06:20:07.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.448+0000 7f9ff4400640 1 -- 172.21.15.163:0/2540672596 wait complete. 2024-07-26T06:20:07.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.450+0000 7f9ff4400640 1 Processor -- start 2024-07-26T06:20:07.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.450+0000 7f9ff4400640 1 -- start start 2024-07-26T06:20:07.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff4400640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff4400640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0d3030 0x7f9fec15b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff4400640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec15d540 0x7f9fec0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff4400640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec0b6be0 con 0x7f9fec0d0370 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff4400640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec0b6a60 con 0x7f9fec15d540 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff4400640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9fec0b6d60 con 0x7f9fec0d3030 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff1a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff1a00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0a9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52636/0 (socket says 172.21.15.163:52636) 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.451+0000 7f9ff1a00640 1 -- 172.21.15.163:0/2632504693 learned_addr learned my addr 172.21.15.163:0/2632504693 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff1a00640 1 -- 172.21.15.163:0/2632504693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0d3030 msgr2=0x7f9fec15b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff1a00640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0d3030 0x7f9fec15b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff1a00640 1 -- 172.21.15.163:0/2632504693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec15d540 msgr2=0x7f9fec0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff1a00640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec15d540 0x7f9fec0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff1a00640 1 -- 172.21.15.163:0/2632504693 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec0cbeb0 con 0x7f9fec0d0370 2024-07-26T06:20:07.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff1a00640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0a9fb0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00cb30 tx=0x7f9fdc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:20:07.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc016020 con 0x7f9fec0d0370 2024-07-26T06:20:07.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fdc00ad90 con 0x7f9fec0d0370 2024-07-26T06:20:07.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.452+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec0cc190 con 0x7f9fec0d0370 2024-07-26T06:20:07.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.453+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc012680 con 0x7f9fec0d0370 2024-07-26T06:20:07.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.453+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec0cc750 con 0x7f9fec0d0370 2024-07-26T06:20:07.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.454+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fec0b6520 con 0x7f9fec0d0370 2024-07-26T06:20:07.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.455+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7f9fdc019020 con 0x7f9fec0d0370 2024-07-26T06:20:07.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.456+0000 7f9fe9600640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9fcc076630 0x7f9fcc078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.456+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7f9fdc0980e0 con 0x7f9fec0d0370 2024-07-26T06:20:07.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.456+0000 7f9ff1000640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9fcc076630 0x7f9fcc078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.457+0000 7f9ff1000640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9fcc076630 0x7f9fcc078af0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8005fd0 tx=0x7f9fd8005850 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:20:07.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.460+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9fdc020020 con 0x7f9fec0d0370 2024-07-26T06:20:07.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.659+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f9fec0b8320 con 0x7f9fec0d0370 2024-07-26T06:20:07.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.664+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v40)=0 v40) ==== 123+0+0 (secure 0 0 0) 0x7f9fdc061930 con 0x7f9fec0d0370 2024-07-26T06:20:07.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.666+0000 7f9fe9600640 1 -- 172.21.15.163:0/2632504693 <== mon.0 v2:172.21.15.163:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f9fdc060d40 con 0x7f9fec0d0370 2024-07-26T06:20:07.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9fcc076630 msgr2=0x7f9fcc078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9fcc076630 0x7f9fcc078af0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8005fd0 tx=0x7f9fd8005850 comp rx=0 tx=0).stop 2024-07-26T06:20:07.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 msgr2=0x7f9fec0a9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0a9fb0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00cb30 tx=0x7f9fdc00a060 comp rx=0 tx=0).stop 2024-07-26T06:20:07.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 shutdown_connections 2024-07-26T06:20:07.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7f9fcc076630 0x7f9fcc078af0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec15d540 0x7f9fec0ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0d3030 0x7f9fec15b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 --2- 172.21.15.163:0/2632504693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fec0d0370 0x7f9fec0a9fb0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 >> 172.21.15.163:0/2632504693 conn(0x7f9fec0b4fe0 msgr2=0x7f9fec0b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:20:07.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 shutdown_connections 2024-07-26T06:20:07.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.668+0000 7f9ff4400640 1 -- 172.21.15.163:0/2632504693 wait complete. 2024-07-26T06:20:07.688 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-07-26T06:20:07.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-07-26T06:20:07.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 -- 172.21.15.163:0/299292490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c0ce260 msgr2=0x7fdc4c15ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 --2- 172.21.15.163:0/299292490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c0ce260 0x7fdc4c15ef70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005fd0 tx=0x7fdc3802a5d0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 -- 172.21.15.163:0/299292490 shutdown_connections 2024-07-26T06:20:07.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 --2- 172.21.15.163:0/299292490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c0ce260 0x7fdc4c15ef70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 --2- 172.21.15.163:0/299292490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc4c15b240 0x7fdc4c0cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 -- 172.21.15.163:0/299292490 >> 172.21.15.163:0/299292490 conn(0x7fdc4c06c820 msgr2=0x7fdc4c0b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:20:07.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 -- 172.21.15.163:0/299292490 shutdown_connections 2024-07-26T06:20:07.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.900+0000 7fdc54200640 1 -- 172.21.15.163:0/299292490 wait complete. 2024-07-26T06:20:07.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.902+0000 7fdc54200640 1 Processor -- start 2024-07-26T06:20:07.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.902+0000 7fdc54200640 1 -- start start 2024-07-26T06:20:07.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.903+0000 7fdc54200640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc4c0ce260 0x7fdc4c15e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.903+0000 7fdc54200640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 0x7fdc4c13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.903+0000 7fdc54200640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c15f4b0 0x7fdc4c142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.903+0000 7fdc54200640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c073e10 con 0x7fdc4c15b240 2024-07-26T06:20:07.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.903+0000 7fdc54200640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c073c90 con 0x7fdc4c15f4b0 2024-07-26T06:20:07.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.903+0000 7fdc54200640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4c073f90 con 0x7fdc4c0ce260 2024-07-26T06:20:07.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 0x7fdc4c13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 0x7fdc4c13cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52642/0 (socket says 172.21.15.163:52642) 2024-07-26T06:20:07.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 -- 172.21.15.163:0/1725227868 learned_addr learned my addr 172.21.15.163:0/1725227868 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:20:07.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc52200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c15f4b0 0x7fdc4c142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc51800640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc4c0ce260 0x7fdc4c15e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 -- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc4c0ce260 msgr2=0x7fdc4c15e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc4c0ce260 0x7fdc4c15e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 -- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c15f4b0 msgr2=0x7fdc4c142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:07.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c15f4b0 0x7fdc4c142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:07.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.904+0000 7fdc50e00640 1 -- 172.21.15.163:0/1725227868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c15c140 con 0x7fdc4c15b240 2024-07-26T06:20:07.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.905+0000 7fdc50e00640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 0x7fdc4c13cfe0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fdc38006010 tx=0x7fdc38003670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.905+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3803f070 con 0x7fdc4c15b240 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.905+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fdc38003d80 con 0x7fdc4c15b240 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.905+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38035030 con 0x7fdc4c15b240 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.905+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4c15c420 con 0x7fdc4c15b240 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.905+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc4c15c9e0 con 0x7fdc4c15b240 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.906+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4c15ef70 con 0x7fdc4c15b240 2024-07-26T06:20:07.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.908+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 20) ==== 98507+0+0 (secure 0 0 0) 0x7fdc38042070 con 0x7fdc4c15b240 2024-07-26T06:20:07.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.909+0000 7fdc49600640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc2c076630 0x7fdc2c078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:20:07.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.909+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4943+0+0 (secure 0 0 0) 0x7fdc380b8680 con 0x7fdc4c15b240 2024-07-26T06:20:07.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.909+0000 7fdc51800640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc2c076630 0x7fdc2c078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:20:07.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.909+0000 7fdc51800640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc2c076630 0x7fdc2c078af0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c15f230 tx=0x7fdc40005f90 comp rx=0 tx=0).ready entity=mgr.4118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:20:07.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:07.913+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc38081d80 con 0x7fdc4c15b240 2024-07-26T06:20:08.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.154+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc4c06ee20 con 0x7fdc4c15b240 2024-07-26T06:20:08.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.155+0000 7fdc49600640 1 -- 172.21.15.163:0/1725227868 <== mon.0 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+3920 (secure 0 0 0) 0x7fdc38081680 con 0x7fdc4c15b240 2024-07-26T06:20:08.156 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:20:08.156 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":73,"btime":"2024-07-26T06:19:26:766238+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6834/3792489553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3792489553},{"type":"v1","addr":"172.21.15.164:6834","nonce":3792489553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5309,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/664133269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6817","nonce":664133269},{"type":"v1","addr":"172.21.15.164:6835","nonce":664133269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":5620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/4156461040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6833","nonce":4156461040},{"type":"v1","addr":"172.21.15.163:6835","nonce":4156461040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65},{"gid":7191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6834/2493773848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":2493773848},{"type":"v1","addr":"172.21.15.163:6834","nonce":2493773848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[]} 2024-07-26T06:20:08.157 INFO:teuthology.orchestra.run.smithi163.stderr:dumped fsmap epoch 73 2024-07-26T06:20:08.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc2c076630 msgr2=0x7fdc2c078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:08.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc2c076630 0x7fdc2c078af0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c15f230 tx=0x7fdc40005f90 comp rx=0 tx=0).stop 2024-07-26T06:20:08.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 msgr2=0x7fdc4c13cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 0x7fdc4c13cfe0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fdc38006010 tx=0x7fdc38003670 comp rx=0 tx=0).stop 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 shutdown_connections 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:6836/3263392688,v1:172.21.15.163:6837/3263392688] conn(0x7fdc2c076630 0x7fdc2c078af0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc4c15f4b0 0x7fdc4c142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc4c15b240 0x7fdc4c13cfe0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 --2- 172.21.15.163:0/1725227868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc4c0ce260 0x7fdc4c15e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:20:08.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 >> 172.21.15.163:0/1725227868 conn(0x7fdc4c06c820 msgr2=0x7fdc4c073580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:20:08.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 shutdown_connections 2024-07-26T06:20:08.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:20:08.158+0000 7fdc54200640 1 -- 172.21.15.163:0/1725227868 wait complete. 2024-07-26T06:20:08.178 DEBUG:tasks.cephfs.filesystem:already dead... 2024-07-26T06:20:08.178 INFO:teuthology.misc:Shutting down mds daemons... 2024-07-26T06:20:08.179 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-07-26T06:20:08.179 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.201 INFO:tasks.ceph.mds.a:Stopped 2024-07-26T06:20:08.201 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-07-26T06:20:08.201 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.271 INFO:tasks.ceph.mds.c:Stopped 2024-07-26T06:20:08.272 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-07-26T06:20:08.272 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.301 INFO:tasks.ceph.mds.b:Stopped 2024-07-26T06:20:08.301 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-07-26T06:20:08.301 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.367 INFO:tasks.ceph.mds.d:Stopped 2024-07-26T06:20:08.367 INFO:teuthology.misc:Shutting down osd daemons... 2024-07-26T06:20:08.368 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-07-26T06:20:08.368 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.427 INFO:tasks.ceph.osd.0:Stopped 2024-07-26T06:20:08.427 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-07-26T06:20:08.428 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.488 INFO:tasks.ceph.osd.1:Stopped 2024-07-26T06:20:08.489 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-07-26T06:20:08.489 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.542 INFO:tasks.ceph.osd.2:Stopped 2024-07-26T06:20:08.542 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-07-26T06:20:08.542 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.632 INFO:tasks.ceph.osd.3:Stopped 2024-07-26T06:20:08.633 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-07-26T06:20:08.633 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.713 INFO:tasks.ceph.osd.4:Stopped 2024-07-26T06:20:08.714 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-07-26T06:20:08.714 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.802 INFO:tasks.ceph.osd.5:Stopped 2024-07-26T06:20:08.802 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-07-26T06:20:08.802 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.869 INFO:tasks.ceph.osd.6:Stopped 2024-07-26T06:20:08.869 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-07-26T06:20:08.869 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:08.925 INFO:tasks.ceph.osd.7:Stopped 2024-07-26T06:20:08.926 INFO:teuthology.misc:Shutting down mgr daemons... 2024-07-26T06:20:08.927 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-07-26T06:20:08.927 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:09.020 INFO:tasks.ceph.mgr.y:Stopped 2024-07-26T06:20:09.020 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-07-26T06:20:09.020 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:09.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-07-26T06:20:14.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2024-07-26T06:20:15.122 INFO:tasks.ceph.mgr.x:Stopped 2024-07-26T06:20:15.123 INFO:teuthology.misc:Shutting down mon daemons... 2024-07-26T06:20:15.123 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-07-26T06:20:15.123 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:15.161 INFO:tasks.ceph.mon.a:Stopped 2024-07-26T06:20:15.162 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-07-26T06:20:15.162 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:15.210 INFO:tasks.ceph.mon.b:Stopped 2024-07-26T06:20:15.210 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-07-26T06:20:15.211 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T06:20:15.260 INFO:tasks.ceph.mon.c:Stopped 2024-07-26T06:20:15.261 INFO:tasks.ceph:Checking cluster log for badness... 2024-07-26T06:20:15.261 DEBUG:teuthology.orchestra.run.smithi163:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-07-26T06:20:15.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:20:15.373 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-07-26T06:20:15.539 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:20:15.539 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-07-26T06:20:15.691 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:20:15.692 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-07-26T06:20:15.840 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:20:15.841 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-07-26T06:20:15.991 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:20:15.992 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-07-26T06:20:16.217 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:20:16.217 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-07-26T06:20:16.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:20:16.366 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-07-26T06:20:16.534 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:20:16.534 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-07-26T06:20:16.694 INFO:tasks.ceph:Archiving mon data... 2024-07-26T06:20:16.695 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/data/mon.a.tgz 2024-07-26T06:20:16.696 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-07-26T06:20:16.752 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.BGtvbxQWQR 2024-07-26T06:20:16.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BGtvbxQWQR 2024-07-26T06:20:17.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.BGtvbxQWQR 2024-07-26T06:20:17.183 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.BGtvbxQWQR is 1MB 2024-07-26T06:20:17.302 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.BGtvbxQWQR 2024-07-26T06:20:17.358 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/data/mon.b.tgz 2024-07-26T06:20:17.359 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-07-26T06:20:17.417 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.yWkqfBBxQA 2024-07-26T06:20:17.417 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.yWkqfBBxQA 2024-07-26T06:20:17.787 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.yWkqfBBxQA 2024-07-26T06:20:17.918 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.yWkqfBBxQA is 1MB 2024-07-26T06:20:18.044 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.yWkqfBBxQA 2024-07-26T06:20:18.211 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/data/mon.c.tgz 2024-07-26T06:20:18.212 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-07-26T06:20:18.263 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.jNVRm1OmfT 2024-07-26T06:20:18.264 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.jNVRm1OmfT 2024-07-26T06:20:18.621 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.jNVRm1OmfT 2024-07-26T06:20:18.762 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.jNVRm1OmfT is 1MB 2024-07-26T06:20:18.884 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.jNVRm1OmfT 2024-07-26T06:20:18.942 INFO:tasks.ceph:Cleaning ceph cluster... 2024-07-26T06:20:18.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-07-26T06:20:18.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-07-26T06:20:19.088 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-07-26T06:20:19.143 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-07-26T06:20:19.143 INFO:tasks.ceph:Archiving crash dumps... 2024-07-26T06:20:19.146 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/remote/smithi163/crash 2024-07-26T06:20:19.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-07-26T06:20:19.237 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/remote/smithi164/crash 2024-07-26T06:20:19.238 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-07-26T06:20:19.324 INFO:tasks.ceph:Shutting down logrotate 2024-07-26T06:20:19.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:20:19.332 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-07-26T06:20:19.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-07-26T06:20:19.540 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-07-26T06:20:19.631 INFO:tasks.ceph:Compressing logs... 2024-07-26T06:20:19.631 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-07-26T06:20:19.639 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-07-26T06:20:19.715 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45905.log 2024-07-26T06:20:19.716 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43003.log 2024-07-26T06:20:19.717 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55166.log 2024-07-26T06:20:19.717 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45905.log.gz 2024-07-26T06:20:19.718 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64775.log 2024-07-26T06:20:19.718 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.43003.log.gz 2024-07-26T06:20:19.719 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45643.log 2024-07-26T06:20:19.720 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55166.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55166.log.gz 2024-07-26T06:20:19.720 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43727.log 2024-07-26T06:20:19.721 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44703.log 2024-07-26T06:20:19.721 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.45643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45643.log.gz 2024-07-26T06:20:19.721 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44158.log 2024-07-26T06:20:19.722 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2024-07-26T06:20:19.722 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64775.log.gz 2024-07-26T06:20:19.722 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43727.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43727.log.gz 2024-07-26T06:20:19.722 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53800.log 2024-07-26T06:20:19.724 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50908.log 2024-07-26T06:20:19.725 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2024-07-26T06:20:19.725 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.53800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53800.log.gz 2024-07-26T06:20:19.725 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43375.log 2024-07-26T06:20:19.725 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.44703.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.1.44703.log.gz 2024-07-26T06:20:19.726 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-07-26T06:20:19.726 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.44158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44158.log.gz 2024-07-26T06:20:19.726 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44707.log 2024-07-26T06:20:19.726 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46761.log 2024-07-26T06:20:19.726 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.44707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44707.log.gz 2024-07-26T06:20:19.726 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-07-26T06:20:19.728 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.46761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46761.log.gz 2024-07-26T06:20:19.728 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-07-26T06:20:19.728 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-07-26T06:20:19.728 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47259.log 2024-07-26T06:20:19.729 INFO:teuthology.orchestra.run.smithi164.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-07-26T06:20:19.730 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57835.log 2024-07-26T06:20:19.730 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50908.log.gz 2024-07-26T06:20:19.730 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49513.log 2024-07-26T06:20:19.730 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43375.log.gz 2024-07-26T06:20:19.730 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58129.log 2024-07-26T06:20:19.731 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57835.log.gz 2024-07-26T06:20:19.731 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49513.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.49513.log.gz 2024-07-26T06:20:19.731 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52343.log 2024-07-26T06:20:19.731 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37795.log 2024-07-26T06:20:19.731 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58129.log.gz 2024-07-26T06:20:19.733 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43171.log 2024-07-26T06:20:19.734 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52343.log.gz 2024-07-26T06:20:19.734 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37795.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37795.log.gz 2024-07-26T06:20:19.734 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50824.log 2024-07-26T06:20:19.734 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49853.log 2024-07-26T06:20:19.734 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43171.log.gz 2024-07-26T06:20:19.735 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37969.log 2024-07-26T06:20:19.735 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.47259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47259.log.gz 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37969.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37969.log.gz 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43583.log 2024-07-26T06:20:19.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43165.log 2024-07-26T06:20:19.738 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.3.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-07-26T06:20:19.738 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.43583.log.gz 2024-07-26T06:20:19.739 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56834.log 2024-07-26T06:20:19.739 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50824.log.gz 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49940.log 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49853.log.gz 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46946.log 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56834.log.gz 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64691.log 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49940.log.gz 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46946.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.46946.log.gz 2024-07-26T06:20:19.740 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61905.log 2024-07-26T06:20:19.742 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.43165.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.1.43165.log.gz -- 2024-07-26T06:20:19.742 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-mon.c.log 2024-07-26T06:20:19.742 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47263.log 2024-07-26T06:20:19.743 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-07-26T06:20:19.743 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42652.log 2024-07-26T06:20:19.744 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.64691.log.gz 2024-07-26T06:20:19.744 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59583.log 2024-07-26T06:20:19.744 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61905.log.gz 2024-07-26T06:20:19.744 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58937.log 2024-07-26T06:20:19.744 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42652.log.gz 2024-07-26T06:20:19.744 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62303.log 2024-07-26T06:20:19.745 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.59583.log.gz 2024-07-26T06:20:19.746 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.47263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.47263.log.gz 2024-07-26T06:20:19.746 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45639.log 2024-07-26T06:20:19.747 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62777.log 2024-07-26T06:20:19.747 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58937.log.gz 2024-07-26T06:20:19.747 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51870.log 2024-07-26T06:20:19.747 INFO:teuthology.orchestra.run.smithi163.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.62303.log.gz 2024-07-26T06:20:19.747 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62777.log: gzip 86.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.62777.log.gz --verbose 2024-07-26T06:20:19.747 INFO:teuthology.orchestra.run.smithi163.stderr: -- /var/log/ceph/ceph-client.admin.61272.log 2024-07-26T06:20:19.748 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46180.log 2024-07-26T06:20:19.748 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53126.log 2024-07-26T06:20:19.749 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51870.log.gz 2024-07-26T06:20:19.751 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61272.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61272.log.gz 2024-07-26T06:20:19.751 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49387.log 2024-07-26T06:20:19.751 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50320.log 2024-07-26T06:20:19.751 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53126.log.gz 2024-07-26T06:20:19.752 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.45639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.45639.log.gz 2024-07-26T06:20:19.752 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2024-07-26T06:20:19.752 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.46180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46180.log.gz 2024-07-26T06:20:19.752 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38209.log 2024-07-26T06:20:19.754 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43395.log 2024-07-26T06:20:19.754 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49387.log.gz 2024-07-26T06:20:19.754 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2024-07-26T06:20:19.754 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50320.log.gz 2024-07-26T06:20:19.754 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55940.log 2024-07-26T06:20:19.754 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43395.log.gz 2024-07-26T06:20:19.755 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38129.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2024-07-26T06:20:19.755 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43579.log 2024-07-26T06:20:19.756 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58997.log 2024-07-26T06:20:19.756 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2024-07-26T06:20:19.756 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55940.log: gzip -5 87.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.55940.log.gz -- 2024-07-26T06:20:19.756 INFO:teuthology.orchestra.run.smithi163.stderr: /var/log/ceph/ceph-client.admin.37758.log 2024-07-26T06:20:19.758 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2024-07-26T06:20:19.758 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58997.log.gz 2024-07-26T06:20:19.759 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46988.log 2024-07-26T06:20:19.759 INFO:teuthology.orchestra.run.smithi163.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37758.log.gz 2024-07-26T06:20:19.759 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44162.log 2024-07-26T06:20:19.759 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.3.43579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.43579.log.gz 2024-07-26T06:20:19.759 INFO:teuthology.orchestra.run.smithi164.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38209.log.gz 2024-07-26T06:20:19.760 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60900.log 2024-07-26T06:20:19.761 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gz 2024-07-26T06:20:19.761 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38442.log 2024-07-26T06:20:19.761 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46988.log.gz 2024-07-26T06:20:19.762 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46387.log 2024-07-26T06:20:19.762 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60900.log.gz 2024-07-26T06:20:19.763 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.44162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44162.log.gz 2024-07-26T06:20:19.763 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38009.log 2024-07-26T06:20:19.764 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.38442.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz -5 2024-07-26T06:20:19.764 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46742.log 2024-07-26T06:20:19.764 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56538.log 2024-07-26T06:20:19.764 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2024-07-26T06:20:19.766 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38049.log 2024-07-26T06:20:19.766 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45821.log 2024-07-26T06:20:19.766 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46742.log.gz 2024-07-26T06:20:19.767 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.56538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56538.log.gz 2024-07-26T06:20:19.767 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44557.log 2024-07-26T06:20:19.769 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42919.log 2024-07-26T06:20:19.769 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45821.log.gz 2024-07-26T06:20:19.769 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51060.log 2024-07-26T06:20:19.769 INFO:teuthology.orchestra.run.smithi163.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44557.log.gz 2024-07-26T06:20:19.769 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38049.log: /var/log/ceph/ceph-client.admin.38009.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.38009.log.gz 2024-07-26T06:20:19.770 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38089.log 2024-07-26T06:20:19.771 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43184.log 2024-07-26T06:20:19.772 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56718.log 2024-07-26T06:20:19.772 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42919.log.gz 2024-07-26T06:20:19.772 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47569.log 2024-07-26T06:20:19.772 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51060.log.gz 2024-07-26T06:20:19.772 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48253.log 2024-07-26T06:20:19.772 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56718.log.gz 2024-07-26T06:20:19.774 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8%/var/log/ceph/ceph-client.admin.38089.log: -- replaced with /var/log/ceph/ceph-client.admin.38049.log.gz 2024-07-26T06:20:19.774 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53504.log 2024-07-26T06:20:19.775 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47569.log.gz 2024-07-26T06:20:19.775 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58685.log 2024-07-26T06:20:19.775 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48253.log.gz 2024-07-26T06:20:19.775 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37727.log 2024-07-26T06:20:19.775 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53504.log.gz 2024-07-26T06:20:19.777 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48295.log 2024-07-26T06:20:19.778 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58685.log.gz 2024-07-26T06:20:19.778 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61827.log 2024-07-26T06:20:19.778 INFO:teuthology.orchestra.run.smithi163.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37727.log.gz 2024-07-26T06:20:19.779 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.43184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46757.log 2024-07-26T06:20:19.779 INFO:teuthology.orchestra.run.smithi164.stderr: 86.2% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38089.log.gz 2024-07-26T06:20:19.779 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.1.43184.log.gz 2024-07-26T06:20:19.779 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46184.log 2024-07-26T06:20:19.781 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45509.log 2024-07-26T06:20:19.781 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48295.log.gz 2024-07-26T06:20:19.781 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.61827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61827.log.gz 2024-07-26T06:20:19.781 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37359.log 2024-07-26T06:20:19.782 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.2.45509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45509.log.gz 2024-07-26T06:20:19.782 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46597.log 2024-07-26T06:20:19.782 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37359.log.gz 2024-07-26T06:20:19.782 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63501.log 2024-07-26T06:20:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61527.log 2024-07-26T06:20:19.783 INFO:teuthology.orchestra.run.smithi163.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46597.log.gz 2024-07-26T06:20:19.785 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.46184.log: /var/log/ceph/ceph-client.1.46757.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-07-26T06:20:19.785 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46757.log.gz 2024-07-26T06:20:19.785 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-07-26T06:20:19.787 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63501.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63501.log.gz 2024-07-26T06:20:19.787 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58727.log 2024-07-26T06:20:19.787 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61527.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61527.log.gz 2024-07-26T06:20:19.787 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53084.log 2024-07-26T06:20:19.787 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58727.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-07-26T06:20:19.787 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58727.log.gz 2024-07-26T06:20:19.788 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62844.log 2024-07-26T06:20:19.788 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53084.log.gz 2024-07-26T06:20:19.789 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45863.log 2024-07-26T06:20:19.790 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.46184.log.gz 2024-07-26T06:20:19.790 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38249.log 2024-07-26T06:20:19.791 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51786.log 2024-07-26T06:20:19.791 INFO:teuthology.orchestra.run.smithi163.stderr: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.62844.log.gz 2024-07-26T06:20:19.791 INFO:teuthology.orchestra.run.smithi163.stderr: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-07-26T06:20:19.791 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59501.log 2024-07-26T06:20:19.791 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45863.log.gz 2024-07-26T06:20:19.792 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61400.log 2024-07-26T06:20:19.792 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51786.log.gz 2024-07-26T06:20:19.793 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57877.log 2024-07-26T06:20:19.793 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59501.log.gz 2024-07-26T06:20:19.794 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61400.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61400.log.gz 2024-07-26T06:20:19.794 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51144.log 2024-07-26T06:20:19.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-07-26T06:20:19.795 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58769.log 2024-07-26T06:20:19.795 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57877.log.gz 2024-07-26T06:20:19.796 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38249.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38249.log.gz 2024-07-26T06:20:19.797 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60985.log 2024-07-26T06:20:19.797 INFO:teuthology.orchestra.run.smithi163.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.51144.log.gz 2024-07-26T06:20:19.797 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58769.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.58769.log.gz 2024-07-26T06:20:19.798 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55292.log 2024-07-26T06:20:19.799 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50152.log 2024-07-26T06:20:19.799 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60985.log.gz 2024-07-26T06:20:19.800 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59105.log 2024-07-26T06:20:19.800 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55292.log.gz 2024-07-26T06:20:19.801 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53336.log 2024-07-26T06:20:19.801 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50152.log.gz 2024-07-26T06:20:19.802 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59105.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59105.log.gz 2024-07-26T06:20:19.802 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-07-26T06:20:19.803 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43045.log 2024-07-26T06:20:19.804 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53336.log.gz 2024-07-26T06:20:19.804 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37891.log 2024-07-26T06:20:19.806 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54662.log 2024-07-26T06:20:19.806 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43045.log.gz 2024-07-26T06:20:19.807 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50782.log 2024-07-26T06:20:19.807 INFO:teuthology.orchestra.run.smithi163.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37891.log.gz 2024-07-26T06:20:19.808 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52750.log 2024-07-26T06:20:19.808 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54662.log.gz 2024-07-26T06:20:19.809 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37528.log 2024-07-26T06:20:19.809 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50782.log.gz 2024-07-26T06:20:19.810 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60224.log 2024-07-26T06:20:19.810 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52750.log.gz 2024-07-26T06:20:19.811 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37528.log: 77.6% -- replaced with /var/log/ceph/ceph-client.admin.37528.log.gz 2024-07-26T06:20:19.811 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64113.log 2024-07-26T06:20:19.812 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46555.log 2024-07-26T06:20:19.813 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.60224.log.gz 2024-07-26T06:20:19.813 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64239.log 2024-07-26T06:20:19.813 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64113.log.gz 2024-07-26T06:20:19.814 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46555.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46555.log.gz 2024-07-26T06:20:19.815 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54746.log 2024-07-26T06:20:19.816 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64239.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.64239.log.gz 2024-07-26T06:20:19.816 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51372.log 2024-07-26T06:20:19.816 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63861.log 2024-07-26T06:20:19.817 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54746.log.gz 2024-07-26T06:20:19.818 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61484.log 2024-07-26T06:20:19.818 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51372.log.gz 2024-07-26T06:20:19.819 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47779.log 2024-07-26T06:20:19.819 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63861.log.gz 2024-07-26T06:20:19.820 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56912.log 2024-07-26T06:20:19.820 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61484.log.gz 2024-07-26T06:20:19.821 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63086.log 2024-07-26T06:20:19.821 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47779.log.gz 2024-07-26T06:20:19.822 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43667.log 2024-07-26T06:20:19.822 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56912.log.gz 2024-07-26T06:20:19.823 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63254.log 2024-07-26T06:20:19.823 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63086.log.gz 2024-07-26T06:20:19.824 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48121.log 2024-07-26T06:20:19.824 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2024-07-26T06:20:19.825 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61142.log 2024-07-26T06:20:19.825 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63254.log.gz 2024-07-26T06:20:19.826 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.48121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48121.log.gz 2024-07-26T06:20:19.826 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60560.log 2024-07-26T06:20:19.827 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50026.log 2024-07-26T06:20:19.827 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61142.log.gz 2024-07-26T06:20:19.828 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43689.log 2024-07-26T06:20:19.828 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60560.log.gz 2024-07-26T06:20:19.829 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46639.log 2024-07-26T06:20:19.829 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50026.log.gz 2024-07-26T06:20:19.830 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43689.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43689.log.gz 2024-07-26T06:20:19.831 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51600.log 2024-07-26T06:20:19.831 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46639.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-07-26T06:20:19.832 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46639.log.gz 2024-07-26T06:20:19.833 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64575.log 2024-07-26T06:20:19.833 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51600.log.gz 2024-07-26T06:20:19.834 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37925.log 2024-07-26T06:20:19.835 INFO:teuthology.orchestra.run.smithi164.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-07-26T06:20:19.835 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64575.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64575.log.gz 2024-07-26T06:20:19.835 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46100.log 2024-07-26T06:20:19.836 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37925.log: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-07-26T06:20:19.837 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56112.log 2024-07-26T06:20:19.837 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37925.log.gz 2024-07-26T06:20:19.837 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51186.log 2024-07-26T06:20:19.838 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46100.log.gz 2024-07-26T06:20:19.838 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63543.log 2024-07-26T06:20:19.839 INFO:teuthology.orchestra.run.smithi163.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.56112.log.gz 2024-07-26T06:20:19.839 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51186.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51186.log.gz 2024-07-26T06:20:19.840 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52953.log 2024-07-26T06:20:19.841 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45182.log 2024-07-26T06:20:19.841 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63543.log.gz 2024-07-26T06:20:19.841 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47030.log 2024-07-26T06:20:19.842 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52953.log.gz 2024-07-26T06:20:19.843 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44068.log 2024-07-26T06:20:19.843 INFO:teuthology.orchestra.run.smithi163.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45182.log.gz 2024-07-26T06:20:19.844 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47030.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47030.log.gz 2024-07-26T06:20:19.844 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58601.log 2024-07-26T06:20:19.845 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54998.log 2024-07-26T06:20:19.845 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44068.log.gz 2024-07-26T06:20:19.846 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58601.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58601.log.gz 2024-07-26T06:20:19.846 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48555.log 2024-07-26T06:20:19.847 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43732.log 2024-07-26T06:20:19.847 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54998.log.gz 2024-07-26T06:20:19.848 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2024-07-26T06:20:19.848 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.48555.log.gz 2024-07-26T06:20:19.849 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37911.log 2024-07-26T06:20:19.849 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43732.log.gz 2024-07-26T06:20:19.850 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44263.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2024-07-26T06:20:19.850 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60942.log 2024-07-26T06:20:19.850 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46784.log 2024-07-26T06:20:19.851 INFO:teuthology.orchestra.run.smithi163.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37911.log.gz 2024-07-26T06:20:19.852 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51828.log 2024-07-26T06:20:19.852 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60942.log.gz 2024-07-26T06:20:19.853 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59293.log 2024-07-26T06:20:19.853 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46784.log.gz 2024-07-26T06:20:19.854 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51828.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.51828.log.gz 2024-07-26T06:20:19.854 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61047.log 2024-07-26T06:20:19.854 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.59293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59293.log.gz 2024-07-26T06:20:19.855 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51288.log 2024-07-26T06:20:19.855 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52427.log 2024-07-26T06:20:19.856 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61047.log.gz 2024-07-26T06:20:19.856 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54956.log 2024-07-26T06:20:19.857 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51288.log.gz 2024-07-26T06:20:19.858 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52427.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.52427.log.gz 2024-07-26T06:20:19.858 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54353.log 2024-07-26T06:20:19.859 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57164.log 2024-07-26T06:20:19.859 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54956.log.gz 2024-07-26T06:20:19.859 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57427.log 2024-07-26T06:20:19.860 INFO:teuthology.orchestra.run.smithi163.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54353.log.gz 2024-07-26T06:20:19.860 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57164.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57164.log.gz 2024-07-26T06:20:19.861 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53040.log 2024-07-26T06:20:19.862 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53420.log 2024-07-26T06:20:19.862 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57427.log.gz 2024-07-26T06:20:19.863 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53040.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53040.log.gz 2024-07-26T06:20:19.863 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55124.log 2024-07-26T06:20:19.863 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63170.log 2024-07-26T06:20:19.864 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53420.log.gz 2024-07-26T06:20:19.864 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-07-26T06:20:19.865 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55124.log.gz 2024-07-26T06:20:19.866 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59417.log 2024-07-26T06:20:19.866 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63170.log.gz 2024-07-26T06:20:19.866 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42346.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-07-26T06:20:19.867 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44431.log 2024-07-26T06:20:19.867 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44066.log 2024-07-26T06:20:19.868 INFO:teuthology.orchestra.run.smithi163.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.59417.log.gz 2024-07-26T06:20:19.868 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49098.log 2024-07-26T06:20:19.869 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44431.log.gz 2024-07-26T06:20:19.869 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47300.log 2024-07-26T06:20:19.870 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44066.log.gz 2024-07-26T06:20:19.870 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49098.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49098.log.gz 2024-07-26T06:20:19.871 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62387.log 2024-07-26T06:20:19.871 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47300.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-07-26T06:20:19.872 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47300.log.gz 2024-07-26T06:20:19.872 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43390.log 2024-07-26T06:20:19.873 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.62387.log.gz 2024-07-26T06:20:19.873 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2024-07-26T06:20:19.874 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43390.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.43390.log.gz 2024-07-26T06:20:19.875 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52996.log 2024-07-26T06:20:19.875 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54105.log 2024-07-26T06:20:19.876 INFO:teuthology.orchestra.run.smithi163.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2024-07-26T06:20:19.877 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52996.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52996.log.gz 2024-07-26T06:20:19.877 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51414.log 2024-07-26T06:20:19.878 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54105.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54105.log.gz 2024-07-26T06:20:19.878 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63987.log 2024-07-26T06:20:19.879 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51414.log: gzip 87.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.51414.log.gz 2024-07-26T06:20:19.879 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55761.log 2024-07-26T06:20:19.880 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58304.log 2024-07-26T06:20:19.880 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63987.log.gz 2024-07-26T06:20:19.881 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55761.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.55761.log.gz 2024-07-26T06:20:19.881 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60476.log 2024-07-26T06:20:19.882 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47114.log 2024-07-26T06:20:19.882 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58304.log.gz 2024-07-26T06:20:19.883 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60392.log 2024-07-26T06:20:19.883 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60476.log.gz 2024-07-26T06:20:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54395.log 2024-07-26T06:20:19.885 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47114.log.gz 2024-07-26T06:20:19.886 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57690.log 2024-07-26T06:20:19.886 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60392.log.gz 2024-07-26T06:20:19.887 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47481.log 2024-07-26T06:20:19.887 INFO:teuthology.orchestra.run.smithi163.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.54395.log.gz 2024-07-26T06:20:19.888 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60739.log 2024-07-26T06:20:19.888 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57690.log.gz 2024-07-26T06:20:19.889 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44047.log 2024-07-26T06:20:19.889 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47481.log.gz 2024-07-26T06:20:19.890 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60739.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60739.log.gz -5 2024-07-26T06:20:19.891 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64071.log 2024-07-26T06:20:19.892 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44473.log 2024-07-26T06:20:19.892 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44047.log.gz 2024-07-26T06:20:19.893 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57961.log 2024-07-26T06:20:19.893 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64071.log.gz 2024-07-26T06:20:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63585.log 2024-07-26T06:20:19.894 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44473.log.gz 2024-07-26T06:20:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43380.log 2024-07-26T06:20:19.895 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57961.log.gz 2024-07-26T06:20:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63585.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-07-26T06:20:19.896 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63585.log.gz 2024-07-26T06:20:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43380.log: gzip 86.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.43380.log.gz --verbose 2024-07-26T06:20:19.898 INFO:teuthology.orchestra.run.smithi163.stderr: -- /var/log/ceph/ceph-client.admin.52007.log 2024-07-26T06:20:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44017.log 2024-07-26T06:20:19.900 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43389.log 2024-07-26T06:20:19.900 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52007.log.gz 2024-07-26T06:20:19.902 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44305.log 2024-07-26T06:20:19.902 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44017.log.gz 2024-07-26T06:20:19.903 INFO:teuthology.orchestra.run.smithi163.stderr:gzip/var/log/ceph/ceph-client.admin.43389.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46345.log 2024-07-26T06:20:19.903 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43389.log.gz 2024-07-26T06:20:19.904 INFO:teuthology.orchestra.run.smithi164.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-07-26T06:20:19.905 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58895.log 2024-07-26T06:20:19.905 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.44305.log.gz 2024-07-26T06:20:19.906 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50362.log 2024-07-26T06:20:19.906 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46345.log.gz 2024-07-26T06:20:19.907 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37611.log 2024-07-26T06:20:19.907 INFO:teuthology.orchestra.run.smithi163.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.58895.log.gz 2024-07-26T06:20:19.908 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43087.log 2024-07-26T06:20:19.909 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50362.log.gz 2024-07-26T06:20:19.910 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50278.log 2024-07-26T06:20:19.910 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37611.log.gz 2024-07-26T06:20:19.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47821.log 2024-07-26T06:20:19.911 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43087.log.gz 2024-07-26T06:20:19.912 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50278.log: gzip -5 --verbose 87.2% -- /var/log/ceph/ceph-client.admin.50982.log -- replaced with /var/log/ceph/ceph-client.admin.50278.log.gz 2024-07-26T06:20:19.912 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-07-26T06:20:19.913 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59836.log 2024-07-26T06:20:19.914 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47821.log.gz 2024-07-26T06:20:19.915 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52259.log 2024-07-26T06:20:19.915 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50982.log.gz 2024-07-26T06:20:19.916 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-07-26T06:20:19.916 INFO:teuthology.orchestra.run.smithi163.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59836.log.gz 2024-07-26T06:20:19.917 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53882.log 2024-07-26T06:20:19.918 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52259.log.gz 2024-07-26T06:20:19.919 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46178.log 2024-07-26T06:20:19.919 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-07-26T06:20:19.920 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53882.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-07-26T06:20:19.920 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53882.log.gz 2024-07-26T06:20:19.921 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46178.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-07-26T06:20:19.922 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46178.log.gz 2024-07-26T06:20:19.923 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53252.log 2024-07-26T06:20:19.924 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55590.log 2024-07-26T06:20:19.926 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64197.log 2024-07-26T06:20:19.926 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53252.log.gz 2024-07-26T06:20:19.927 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55590.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55590.log.gz 2024-07-26T06:20:19.928 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64197.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64197.log.gz 2024-07-26T06:20:19.929 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37914.log 2024-07-26T06:20:19.930 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64281.log 2024-07-26T06:20:19.932 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59297.log 2024-07-26T06:20:19.932 INFO:teuthology.orchestra.run.smithi163.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.37914.log.gz 2024-07-26T06:20:19.933 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37655.log 2024-07-26T06:20:19.933 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64281.log.gz 2024-07-26T06:20:19.934 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.59297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59297.log.gz 2024-07-26T06:20:19.934 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62938.log 2024-07-26T06:20:19.935 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37655.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37655.log.gz 2024-07-26T06:20:19.936 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63338.log 2024-07-26T06:20:19.937 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62938.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.62938.log.gz --verbose 2024-07-26T06:20:19.937 INFO:teuthology.orchestra.run.smithi163.stderr: -- /var/log/ceph/ceph-osd.0.log 2024-07-26T06:20:19.938 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58643.log 2024-07-26T06:20:19.938 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.63338.log.gz 2024-07-26T06:20:19.941 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.58643.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58643.log.gz 2024-07-26T06:20:19.941 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56616.log 2024-07-26T06:20:19.943 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2024-07-26T06:20:19.944 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56616.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56616.log.gz 2024-07-26T06:20:19.946 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51456.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2024-07-26T06:20:19.946 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54604.log 2024-07-26T06:20:19.948 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50704.log 2024-07-26T06:20:19.949 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54604.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54604.log.gz 2024-07-26T06:20:19.950 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50866.log 2024-07-26T06:20:19.950 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.50704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50704.log.gz 2024-07-26T06:20:19.951 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47737.log 2024-07-26T06:20:19.953 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50866.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50866.log.gz 2024-07-26T06:20:19.953 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48972.log 2024-07-26T06:20:19.954 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47737.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47737.log.gz 2024-07-26T06:20:19.955 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37455.log 2024-07-26T06:20:19.956 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48972.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48972.log.gz 2024-07-26T06:20:19.958 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46429.log/var/log/ceph/ceph-client.admin.37455.log: 2024-07-26T06:20:19.958 INFO:teuthology.orchestra.run.smithi163.stderr: 75.3% -- replaced with /var/log/ceph/ceph-client.admin.37455.log.gz 2024-07-26T06:20:19.959 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52217.log 2024-07-26T06:20:19.960 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46429.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46429.log.gz 2024-07-26T06:20:19.961 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47216.log 2024-07-26T06:20:19.962 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52217.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52217.log.gz 2024-07-26T06:20:19.962 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60819.log 2024-07-26T06:20:19.964 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47216.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47216.log.gz 2024-07-26T06:20:19.965 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-07-26T06:20:19.965 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60819.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60819.log.gz 2024-07-26T06:20:19.966 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47523.log 2024-07-26T06:20:19.967 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55675.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-07-26T06:20:19.968 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60014.log 2024-07-26T06:20:19.969 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47523.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47523.log.gz 2024-07-26T06:20:19.970 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48788.log 2024-07-26T06:20:19.971 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60014.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60014.log.gz 2024-07-26T06:20:19.972 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60518.log 2024-07-26T06:20:19.973 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48788.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48788.log.gz 2024-07-26T06:20:19.973 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62345.log 2024-07-26T06:20:19.975 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60518.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.60518.log.gz 2024-07-26T06:20:19.976 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44062.log 2024-07-26T06:20:19.976 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62345.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.62345.log.gz 2024-07-26T06:20:19.977 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45779.log 2024-07-26T06:20:19.979 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44062.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44062.log.gz 2024-07-26T06:20:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45049.log 2024-07-26T06:20:19.980 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45779.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45779.log.gz 2024-07-26T06:20:19.981 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64155.log 2024-07-26T06:20:19.982 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.45049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45049.log.gz 2024-07-26T06:20:19.983 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52911.log 2024-07-26T06:20:19.984 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64155.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64155.log.gz 2024-07-26T06:20:19.985 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44695.log 2024-07-26T06:20:19.985 INFO:teuthology.orchestra.run.smithi164.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-07-26T06:20:19.986 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52911.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52911.log.gz 2024-07-26T06:20:19.987 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44695.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44695.log.gz 2024-07-26T06:20:19.988 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43213.log 2024-07-26T06:20:19.988 INFO:teuthology.orchestra.run.smithi164.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-07-26T06:20:19.989 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60098.log 2024-07-26T06:20:19.991 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43213.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43213.log.gz 2024-07-26T06:20:19.991 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56346.log 2024-07-26T06:20:19.992 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60098.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60098.log.gz 2024-07-26T06:20:19.993 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55082.log 2024-07-26T06:20:19.994 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56346.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56346.log.gz 2024-07-26T06:20:19.995 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43376.log 2024-07-26T06:20:19.996 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55082.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55082.log.gz 2024-07-26T06:20:19.996 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61226.log 2024-07-26T06:20:19.998 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43376.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43376.log.gz 2024-07-26T06:20:19.999 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61226.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61226.log.gz 2024-07-26T06:20:20.000 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49140.log 2024-07-26T06:20:20.001 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52553.log 2024-07-26T06:20:20.002 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49140.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49140.log.gz 2024-07-26T06:20:20.003 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63380.log 2024-07-26T06:20:20.004 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52553.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52553.log.gz 2024-07-26T06:20:20.005 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55805.log 2024-07-26T06:20:20.006 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63380.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63380.log.gz 2024-07-26T06:20:20.007 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55805.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55805.log.gz 2024-07-26T06:20:20.008 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54788.log 2024-07-26T06:20:20.009 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44045.log 2024-07-26T06:20:20.011 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54788.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54788.log.gz 2024-07-26T06:20:20.011 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49984.log 2024-07-26T06:20:20.012 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44045.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44045.log.gz 2024-07-26T06:20:20.013 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42696.log 2024-07-26T06:20:20.014 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49984.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49984.log.gz 2024-07-26T06:20:20.015 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55040.log 2024-07-26T06:20:20.016 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42696.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42696.log.gz 2024-07-26T06:20:20.017 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62165.log 2024-07-26T06:20:20.017 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55040.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55040.log.gz 2024-07-26T06:20:20.019 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62165.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62165.log.gz 2024-07-26T06:20:20.020 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-07-26T06:20:20.020 INFO:teuthology.orchestra.run.smithi164.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-07-26T06:20:20.022 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55717.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-07-26T06:20:20.023 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60182.log 2024-07-26T06:20:20.024 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45737.log 2024-07-26T06:20:20.026 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60182.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60182.log.gz 2024-07-26T06:20:20.026 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51965.log 2024-07-26T06:20:20.027 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45737.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45737.log.gz 2024-07-26T06:20:20.029 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51965.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51965.log.gz 2024-07-26T06:20:20.029 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49896.log 2024-07-26T06:20:20.031 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54830.log 2024-07-26T06:20:20.032 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49896.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49896.log.gz 2024-07-26T06:20:20.033 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-07-26T06:20:20.034 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54830.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.54830.log.gz 2024-07-26T06:20:20.036 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48615.log 2024-07-26T06:20:20.038 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53210.log 2024-07-26T06:20:20.039 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59638.log 2024-07-26T06:20:20.041 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1%/var/log/ceph/ceph-client.admin.59638.log: -- replaced with /var/log/ceph/ceph-client.admin.48615.log.gz 2024-07-26T06:20:20.042 INFO:teuthology.orchestra.run.smithi163.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.59638.log.gz 2024-07-26T06:20:20.043 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52301.log 2024-07-26T06:20:20.044 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53210.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53210.log.gz 2024-07-26T06:20:20.046 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52301.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52301.log.gz 2024-07-26T06:20:20.047 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54208.log 2024-07-26T06:20:20.048 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37913.log 2024-07-26T06:20:20.052 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54208.log: /var/log/ceph/ceph-client.admin.37913.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54208.log.gz 2024-07-26T06:20:20.052 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58811.log 2024-07-26T06:20:20.054 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63128.log 2024-07-26T06:20:20.055 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58811.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58811.log.gz 2024-07-26T06:20:20.056 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54914.log 2024-07-26T06:20:20.056 INFO:teuthology.orchestra.run.smithi163.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37913.log.gz 2024-07-26T06:20:20.056 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63128.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63128.log.gz 2024-07-26T06:20:20.058 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57732.log 2024-07-26T06:20:20.059 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54914.log.gz 2024-07-26T06:20:20.061 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38138.log 2024-07-26T06:20:20.066 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57732.log: /var/log/ceph/ceph-client.admin.38138.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49261.log 2024-07-26T06:20:20.066 INFO:teuthology.orchestra.run.smithi163.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57732.log.gz 2024-07-26T06:20:20.067 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48494.log 2024-07-26T06:20:20.068 INFO:teuthology.orchestra.run.smithi163.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38138.log.gz/var/log/ceph/ceph-client.admin.49261.log: 2024-07-26T06:20:20.071 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3%/var/log/ceph/ceph-client.admin.48494.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.48494.log.gz 2024-07-26T06:20:20.071 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56534.log 2024-07-26T06:20:20.072 INFO:teuthology.orchestra.run.smithi163.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49261.log.gz 2024-07-26T06:20:20.074 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.56534.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31339.log 2024-07-26T06:20:20.075 INFO:teuthology.orchestra.run.smithi163.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56534.log.gz 2024-07-26T06:20:20.076 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53966.log 2024-07-26T06:20:20.080 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.tmp-client.admin.31339.log: /var/log/ceph/ceph-client.admin.53966.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.31339.log.gz 2024-07-26T06:20:20.081 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59459.log 2024-07-26T06:20:20.083 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37545.log 2024-07-26T06:20:20.083 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53966.log.gz 2024-07-26T06:20:20.084 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59459.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59459.log.gz 2024-07-26T06:20:20.088 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37545.log: 81.1% -- replaced with /var/log/ceph/ceph-client.admin.37545.log.gz 2024-07-26T06:20:20.088 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46864.log 2024-07-26T06:20:20.090 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58045.log 2024-07-26T06:20:20.091 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46864.log.gz 2024-07-26T06:20:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50404.log 2024-07-26T06:20:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58045.log: /var/log/ceph/ceph-client.admin.50404.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58045.log.gz 2024-07-26T06:20:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57648.log 2024-07-26T06:20:20.101 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57648.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.50404.log.gz 2024-07-26T06:20:20.101 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42831.log 2024-07-26T06:20:20.102 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57648.log.gz 2024-07-26T06:20:20.104 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42789.log 2024-07-26T06:20:20.107 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42789.log: /var/log/ceph/ceph-client.admin.42831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47695.log 2024-07-26T06:20:20.110 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47863.log 2024-07-26T06:20:20.111 INFO:teuthology.orchestra.run.smithi163.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42831.log.gz 87.8% 2024-07-26T06:20:20.112 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47695.log: -- replaced with /var/log/ceph/ceph-client.admin.42789.log.gz 2024-07-26T06:20:20.117 INFO:teuthology.orchestra.run.smithi163.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.47695.log.gz/var/log/ceph/ceph-client.admin.47863.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.47863.log.gz 2024-07-26T06:20:20.117 INFO:teuthology.orchestra.run.smithi163.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-07-26T06:20:20.117 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61442.log 2024-07-26T06:20:20.118 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-07-26T06:20:20.118 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49730.log 2024-07-26T06:20:20.120 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61442.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61442.log.gz 2024-07-26T06:20:20.121 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49730.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49730.log.gz 2024-07-26T06:20:20.121 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61091.log 2024-07-26T06:20:20.123 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59678.log 2024-07-26T06:20:20.124 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61091.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61091.log.gz 2024-07-26T06:20:20.126 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59678.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.59678.log.gz 2024-07-26T06:20:20.126 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46471.log 2024-07-26T06:20:20.128 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48888.log 2024-07-26T06:20:20.129 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46471.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46471.log.gz 2024-07-26T06:20:20.130 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48888.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48888.log.gz 2024-07-26T06:20:20.131 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47653.log 2024-07-26T06:20:20.134 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47653.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47653.log.gz 2024-07-26T06:20:20.134 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47388.log 2024-07-26T06:20:20.135 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53924.log 2024-07-26T06:20:20.137 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47388.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47388.log.gz 2024-07-26T06:20:20.138 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53924.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53924.log.gz 2024-07-26T06:20:20.139 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48117.log 2024-07-26T06:20:20.141 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.48117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48117.log.gz 2024-07-26T06:20:20.141 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47258.log 2024-07-26T06:20:20.143 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63028.log 2024-07-26T06:20:20.144 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47258.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47258.log.gz 2024-07-26T06:20:20.146 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63028.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63028.log.gz 2024-07-26T06:20:20.146 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52655.log 2024-07-26T06:20:20.148 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53294.log 2024-07-26T06:20:20.149 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52655.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2024-07-26T06:20:20.150 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53294.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53294.log.gz 2024-07-26T06:20:20.151 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59722.log 2024-07-26T06:20:20.153 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59722.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.59722.log.gz 2024-07-26T06:20:20.154 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59039.log 2024-07-26T06:20:20.155 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56760.log 2024-07-26T06:20:20.157 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59039.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.59039.log.gz 2024-07-26T06:20:20.158 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56760.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56760.log.gz 2024-07-26T06:20:20.158 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55334.log 2024-07-26T06:20:20.160 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50194.log 2024-07-26T06:20:20.161 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55334.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55334.log.gz 2024-07-26T06:20:20.162 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50194.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50194.log.gz 2024-07-26T06:20:20.163 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48337.log 2024-07-26T06:20:20.165 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51642.log 2024-07-26T06:20:20.166 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48337.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48337.log.gz 2024-07-26T06:20:20.167 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51642.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51642.log.gz 2024-07-26T06:20:20.168 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52830.log 2024-07-26T06:20:20.170 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60140.log 2024-07-26T06:20:20.171 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52830.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2024-07-26T06:20:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60140.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60140.log.gz 2024-07-26T06:20:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42877.log 2024-07-26T06:20:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2024-07-26T06:20:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42877.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42877.log.gz 2024-07-26T06:20:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51330.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2024-07-26T06:20:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42747.log 2024-07-26T06:20:20.179 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60308.log 2024-07-26T06:20:20.180 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42747.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42747.log.gz 2024-07-26T06:20:20.181 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60308.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.60308.log.gz 2024-07-26T06:20:20.182 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46268.log 2024-07-26T06:20:20.183 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47929.log 2024-07-26T06:20:20.185 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46268.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46268.log.gz 2024-07-26T06:20:20.186 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47929.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47929.log.gz 2024-07-26T06:20:20.186 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60602.log 2024-07-26T06:20:20.189 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43976.log 2024-07-26T06:20:20.189 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60602.log.gz 2024-07-26T06:20:20.190 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58427.log 2024-07-26T06:20:20.192 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43976.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43976.log.gz 2024-07-26T06:20:20.193 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58427.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58427.log.gz 2024-07-26T06:20:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49555.log 2024-07-26T06:20:20.196 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62261.log 2024-07-26T06:20:20.196 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49555.log.gz 2024-07-26T06:20:20.197 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53546.log 2024-07-26T06:20:20.199 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62261.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62261.log.gz 2024-07-26T06:20:20.201 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53546.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53546.log.gz 2024-07-26T06:20:20.201 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44347.log 2024-07-26T06:20:20.203 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54008.log 2024-07-26T06:20:20.204 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44347.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44347.log.gz 2024-07-26T06:20:20.205 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54008.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54008.log.gz 2024-07-26T06:20:20.205 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51516.log 2024-07-26T06:20:20.207 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63296.log 2024-07-26T06:20:20.208 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51516.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51516.log.gz 2024-07-26T06:20:20.209 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62735.log 2024-07-26T06:20:20.211 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60434.log 2024-07-26T06:20:20.212 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4%/var/log/ceph/ceph-client.admin.62735.log: -- replaced with /var/log/ceph/ceph-client.admin.63296.log.gz 2024-07-26T06:20:20.212 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.62735.log.gz 2024-07-26T06:20:20.214 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60434.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.60434.log.gz 2024-07-26T06:20:20.215 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51228.log 2024-07-26T06:20:20.216 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60056.log 2024-07-26T06:20:20.218 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51228.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51228.log.gz 2024-07-26T06:20:20.219 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60056.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60056.log.gz 2024-07-26T06:20:20.219 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37860.log 2024-07-26T06:20:20.221 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57522.log 2024-07-26T06:20:20.222 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37860.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37860.log.gz 2024-07-26T06:20:20.224 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-07-26T06:20:20.224 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57522.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57522.log.gz 2024-07-26T06:20:20.224 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63212.log 2024-07-26T06:20:20.225 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53612.log 2024-07-26T06:20:20.227 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63212.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63212.log.gz 2024-07-26T06:20:20.228 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53612.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53612.log.gz 2024-07-26T06:20:20.229 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2024-07-26T06:20:20.230 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49014.log 2024-07-26T06:20:20.231 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-07-26T06:20:20.232 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55250.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2024-07-26T06:20:20.233 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49014.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49014.log.gz 2024-07-26T06:20:20.233 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48930.log 2024-07-26T06:20:20.235 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37472.log 2024-07-26T06:20:20.236 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48930.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48930.log.gz 2024-07-26T06:20:20.237 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37472.log: 77.1% -- replaced with /var/log/ceph/ceph-client.admin.37472.log.gz 2024-07-26T06:20:20.238 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52595.log 2024-07-26T06:20:20.239 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49056.log 2024-07-26T06:20:20.241 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52595.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52595.log.gz 2024-07-26T06:20:20.242 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49056.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49056.log.gz 2024-07-26T06:20:20.243 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46699.log 2024-07-26T06:20:20.244 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51102.log 2024-07-26T06:20:20.245 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46699.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46699.log.gz 2024-07-26T06:20:20.247 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51102.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51102.log.gz 2024-07-26T06:20:20.247 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45505.log 2024-07-26T06:20:20.249 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61989.log 2024-07-26T06:20:20.250 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.2.45505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45505.log.gz 2024-07-26T06:20:20.251 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61989.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2024-07-26T06:20:20.252 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42566.log 2024-07-26T06:20:20.253 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59972.log 2024-07-26T06:20:20.255 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42566.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42566.log.gz 2024-07-26T06:20:20.256 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59972.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59972.log.gz 2024-07-26T06:20:20.257 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59375.log 2024-07-26T06:20:20.258 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64533.log 2024-07-26T06:20:20.262 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59375.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59375.log.gz 2024-07-26T06:20:20.263 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64533.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64533.log.gz 2024-07-26T06:20:20.263 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57122.log 2024-07-26T06:20:20.263 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-07-26T06:20:20.264 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64617.log 2024-07-26T06:20:20.264 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57122.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57122.log.gz 2024-07-26T06:20:20.266 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64617.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.64617.log.gz 2024-07-26T06:20:20.266 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57793.log 2024-07-26T06:20:20.268 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58559.log 2024-07-26T06:20:20.269 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57793.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.57793.log.gz 2024-07-26T06:20:20.270 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58559.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58559.log.gz 2024-07-26T06:20:20.271 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43393.log 2024-07-26T06:20:20.273 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47611.log 2024-07-26T06:20:20.274 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43393.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43393.log.gz 2024-07-26T06:20:20.275 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47611.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47611.log.gz 2024-07-26T06:20:20.276 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58224.log 2024-07-26T06:20:20.276 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-07-26T06:20:20.277 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56238.log 2024-07-26T06:20:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58224.log.gz 2024-07-26T06:20:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56238.log: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-07-26T06:20:20.280 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56238.log.gz 2024-07-26T06:20:20.281 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-07-26T06:20:20.281 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m0.590s 2024-07-26T06:20:20.281 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m2.730s 2024-07-26T06:20:20.281 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.366s 2024-07-26T06:20:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56070.log 2024-07-26T06:20:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54704.log 2024-07-26T06:20:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56070.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56070.log.gz 2024-07-26T06:20:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43394.log 2024-07-26T06:20:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52133.log 2024-07-26T06:20:20.284 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54704.log.gz 2024-07-26T06:20:20.285 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43394.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43394.log.gz 2024-07-26T06:20:20.285 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2024-07-26T06:20:20.286 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56954.log 2024-07-26T06:20:20.287 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52133.log.gz 2024-07-26T06:20:20.288 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49650.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2024-07-26T06:20:20.288 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45947.log 2024-07-26T06:20:20.289 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56954.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56954.log.gz 2024-07-26T06:20:20.289 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50700.log 2024-07-26T06:20:20.291 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51744.log 2024-07-26T06:20:20.291 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45947.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45947.log.gz 2024-07-26T06:20:20.292 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.50700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50700.log.gz 2024-07-26T06:20:20.292 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56196.log 2024-07-26T06:20:20.293 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51744.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51744.log.gz 2024-07-26T06:20:20.293 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43723.log 2024-07-26T06:20:20.294 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52385.log 2024-07-26T06:20:20.295 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56196.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.56196.log.gz 2024-07-26T06:20:20.295 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43723.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43723.log.gz 2024-07-26T06:20:20.296 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45650.log 2024-07-26T06:20:20.297 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52385.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52385.log.gz 2024-07-26T06:20:20.297 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61358.log 2024-07-26T06:20:20.298 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64733.log 2024-07-26T06:20:20.299 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45650.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45650.log.gz 2024-07-26T06:20:20.300 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61358.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61358.log.gz 2024-07-26T06:20:20.300 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58470.log 2024-07-26T06:20:20.301 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64733.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64733.log.gz 2024-07-26T06:20:20.301 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51558.log 2024-07-26T06:20:20.302 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58470.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58470.log.gz 2024-07-26T06:20:20.303 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58853.log 2024-07-26T06:20:20.304 INFO:teuthology.orchestra.run.smithi163.stderr:gzip/var/log/ceph/ceph-client.admin.51558.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52091.log 2024-07-26T06:20:20.305 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51558.log.gz 2024-07-26T06:20:20.305 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58853.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58853.log.gz 2024-07-26T06:20:20.305 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44389.log 2024-07-26T06:20:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52091.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52091.log.gz 2024-07-26T06:20:20.307 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42479.log 2024-07-26T06:20:20.308 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64449.log 2024-07-26T06:20:20.308 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44389.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44389.log.gz 2024-07-26T06:20:20.309 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42479.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42479.log.gz 2024-07-26T06:20:20.309 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51684.log 2024-07-26T06:20:20.310 INFO:teuthology.orchestra.run.smithi163.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-07-26T06:20:20.310 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64449.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64449.log.gz 2024-07-26T06:20:20.311 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59543.log 2024-07-26T06:20:20.312 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.51684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55429.log 2024-07-26T06:20:20.312 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51684.log.gz 2024-07-26T06:20:20.312 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2024-07-26T06:20:20.313 INFO:teuthology.orchestra.run.smithi163.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.59543.log.gz 2024-07-26T06:20:20.314 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45989.log 2024-07-26T06:20:20.314 INFO:teuthology.orchestra.run.smithi163.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55429.log.gz 2024-07-26T06:20:20.314 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63819.log 2024-07-26T06:20:20.315 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2024-07-26T06:20:20.316 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47174.log 2024-07-26T06:20:20.316 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45989.log.gz 2024-07-26T06:20:20.317 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61947.log 2024-07-26T06:20:20.317 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63819.log.gz 2024-07-26T06:20:20.318 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57564.log 2024-07-26T06:20:20.318 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47174.log.gz 2024-07-26T06:20:20.319 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64029.log 2024-07-26T06:20:20.319 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61947.log.gz 2024-07-26T06:20:20.320 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54311.log 2024-07-26T06:20:20.320 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57564.log.gz 2024-07-26T06:20:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64029.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64029.log.gz 2024-07-26T06:20:20.321 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59914.log 2024-07-26T06:20:20.322 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61569.log 2024-07-26T06:20:20.322 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54311.log.gz 2024-07-26T06:20:20.323 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.59914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46513.log 2024-07-26T06:20:20.323 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59914.log.gz 2024-07-26T06:20:20.324 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43659.log 2024-07-26T06:20:20.325 INFO:teuthology.orchestra.run.smithi163.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61569.log.gz 2024-07-26T06:20:20.325 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.46513.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46513.log.gz 2024-07-26T06:20:20.325 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53462.log 2024-07-26T06:20:20.326 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61823.log 2024-07-26T06:20:20.326 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43659.log.gz 2024-07-26T06:20:20.327 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58514.log 2024-07-26T06:20:20.327 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53462.log.gz 2024-07-26T06:20:20.328 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.61823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61823.log.gz 2024-07-26T06:20:20.328 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37710.log 2024-07-26T06:20:20.329 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52049.log 2024-07-26T06:20:20.329 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58514.log.gz 2024-07-26T06:20:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58087.log 2024-07-26T06:20:20.330 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37710.log.gz 2024-07-26T06:20:20.331 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49429.log 2024-07-26T06:20:20.331 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52049.log.gz 2024-07-26T06:20:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54526.log 2024-07-26T06:20:20.333 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58087.log.gz 2024-07-26T06:20:20.334 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53804.log 2024-07-26T06:20:20.334 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49429.log.gz 2024-07-26T06:20:20.335 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45030.log 2024-07-26T06:20:20.335 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54526.log.gz 2024-07-26T06:20:20.335 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.53804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53804.log.gz 2024-07-26T06:20:20.336 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64323.log 2024-07-26T06:20:20.336 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.0.45030.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.0.45030.log.gz /var/log/ceph/ceph-client.admin.54872.log 2024-07-26T06:20:20.337 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-07-26T06:20:20.337 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56154.log 2024-07-26T06:20:20.338 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64323.log.gz 2024-07-26T06:20:20.339 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60350.log 2024-07-26T06:20:20.339 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54872.log.gz 2024-07-26T06:20:20.340 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57016.log 2024-07-26T06:20:20.340 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.56154.log.gz 2024-07-26T06:20:20.341 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62031.log 2024-07-26T06:20:20.341 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60350.log.gz 2024-07-26T06:20:20.342 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55632.log 2024-07-26T06:20:20.342 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57016.log.gz 2024-07-26T06:20:20.343 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49471.log 2024-07-26T06:20:20.343 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62031.log.gz 2024-07-26T06:20:20.344 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62219.log 2024-07-26T06:20:20.344 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55632.log.gz 2024-07-26T06:20:20.345 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49471.log: gzip -5 --verbose 86.8% -- -- replaced with /var/log/ceph/ceph-client.admin.49471.log.gz /var/log/ceph/ceph-client.admin.55986.log 2024-07-26T06:20:20.345 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-07-26T06:20:20.346 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43129.log 2024-07-26T06:20:20.346 INFO:teuthology.orchestra.run.smithi163.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.62219.log.gz 2024-07-26T06:20:20.347 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55986.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55986.log.gz 2024-07-26T06:20:20.347 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54166.log 2024-07-26T06:20:20.348 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43129.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-07-26T06:20:20.348 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43129.log.gz 2024-07-26T06:20:20.349 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55509.log 2024-07-26T06:20:20.349 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54166.log.gz 2024-07-26T06:20:20.350 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42247.log 2024-07-26T06:20:20.351 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55509.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55509.log.gz 2024-07-26T06:20:20.351 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63777.log 2024-07-26T06:20:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52175.log 2024-07-26T06:20:20.353 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42247.log.gz 2024-07-26T06:20:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63777.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.63777.log.gz 2024-07-26T06:20:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49345.log 2024-07-26T06:20:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52175.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52175.log.gz 2024-07-26T06:20:20.355 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42608.log 2024-07-26T06:20:20.356 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37378.log 2024-07-26T06:20:20.357 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49345.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49345.log.gz 2024-07-26T06:20:20.357 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42608.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42608.log.gz 2024-07-26T06:20:20.358 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56658.log 2024-07-26T06:20:20.358 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.37378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37378.log.gz 2024-07-26T06:20:20.359 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63903.log 2024-07-26T06:20:20.360 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57310.log 2024-07-26T06:20:20.361 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56658.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56658.log.gz 2024-07-26T06:20:20.361 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63903.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63903.log.gz 2024-07-26T06:20:20.362 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45695.log 2024-07-26T06:20:20.362 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57310.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.57310.log.gz 2024-07-26T06:20:20.363 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63945.log 2024-07-26T06:20:20.364 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49303.log 2024-07-26T06:20:20.364 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.45695.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45695.log.gz 2024-07-26T06:20:20.365 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63945.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63945.log.gz 2024-07-26T06:20:20.366 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61316.log 2024-07-26T06:20:20.366 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49303.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49303.log.gz 2024-07-26T06:20:20.367 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50236.log 2024-07-26T06:20:20.368 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61316.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61316.log.gz 2024-07-26T06:20:20.368 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58385.log 2024-07-26T06:20:20.369 INFO:teuthology.orchestra.run.smithi163.stderr:gzip/var/log/ceph/ceph-client.admin.50236.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48846.log 2024-07-26T06:20:20.370 INFO:teuthology.orchestra.run.smithi163.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50236.log.gz 2024-07-26T06:20:20.370 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58385.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58385.log.gz 2024-07-26T06:20:20.371 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57919.log 2024-07-26T06:20:20.372 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47439.log 2024-07-26T06:20:20.372 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.48846.log.gz 2024-07-26T06:20:20.373 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61635.log 2024-07-26T06:20:20.374 INFO:teuthology.orchestra.run.smithi163.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57919.log.gz 2024-07-26T06:20:20.375 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60266.log 2024-07-26T06:20:20.375 INFO:teuthology.orchestra.run.smithi163.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47439.log.gz 2024-07-26T06:20:20.375 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61635.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61635.log.gz 2024-07-26T06:20:20.377 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60266.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60266.log.gz 2024-07-26T06:20:20.378 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53168.log 2024-07-26T06:20:20.378 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55898.log 2024-07-26T06:20:20.380 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42961.log 2024-07-26T06:20:20.380 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53168.log.gz 2024-07-26T06:20:20.381 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52511.log 2024-07-26T06:20:20.381 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55898.log.gz 2024-07-26T06:20:20.382 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.42961.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42961.log.gz 2024-07-26T06:20:20.383 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52511.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52511.log.gz 2024-07-26T06:20:20.384 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53378.log 2024-07-26T06:20:20.385 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56028.log 2024-07-26T06:20:20.386 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.53378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56280.log 2024-07-26T06:20:20.386 INFO:teuthology.orchestra.run.smithi163.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.53378.log.gz 2024-07-26T06:20:20.387 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56028.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56028.log.gz 2024-07-26T06:20:20.389 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.56280.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.56280.log.gz 2024-07-26T06:20:20.390 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50512.log 2024-07-26T06:20:20.390 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43682.log 2024-07-26T06:20:20.392 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50512.log: gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50512.log.gz 2024-07-26T06:20:20.393 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58003.log 2024-07-26T06:20:20.393 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.43682.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43682.log.gz 2024-07-26T06:20:20.394 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48710.log 2024-07-26T06:20:20.394 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.58003.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58003.log.gz 2024-07-26T06:20:20.395 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55856.log 2024-07-26T06:20:20.396 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44515.log 2024-07-26T06:20:20.396 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48710.log.gz 2024-07-26T06:20:20.398 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63441.log 2024-07-26T06:20:20.398 INFO:teuthology.orchestra.run.smithi163.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55856.log.gz 2024-07-26T06:20:20.399 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44515.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44515.log.gz 2024-07-26T06:20:20.399 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57058.log 2024-07-26T06:20:20.400 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61184.log 2024-07-26T06:20:20.401 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.63441.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63441.log.gz 2024-07-26T06:20:20.401 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52469.log 86.9% -- replaced with /var/log/ceph/ceph-client.admin.57058.log.gz 2024-07-26T06:20:20.401 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-07-26T06:20:20.402 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.61184.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61184.log.gz 2024-07-26T06:20:20.403 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47072.log 2024-07-26T06:20:20.404 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.52469.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.52469.log.gz 2024-07-26T06:20:20.404 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49201.log 2024-07-26T06:20:20.405 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.47072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57606.log 2024-07-26T06:20:20.405 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47072.log.gz 2024-07-26T06:20:20.406 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49201.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49201.log.gz 2024-07-26T06:20:20.407 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57606.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57606.log.gz 2024-07-26T06:20:20.408 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60644.log 2024-07-26T06:20:20.409 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49811.log 2024-07-26T06:20:20.410 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.60644.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.60644.log.gz --verbose 2024-07-26T06:20:20.410 INFO:teuthology.orchestra.run.smithi163.stderr: -- /var/log/ceph/ceph-client.admin.54269.log 2024-07-26T06:20:20.412 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.49811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38340.log 2024-07-26T06:20:20.412 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49811.log.gz 2024-07-26T06:20:20.413 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.54269.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54269.log.gz 2024-07-26T06:20:20.414 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.38340.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38340.log.gz 2024-07-26T06:20:20.415 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57469.log 2024-07-26T06:20:20.415 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64407.log 2024-07-26T06:20:20.417 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.57469.log: gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57469.log.gz 2024-07-26T06:20:20.417 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55208.log 2024-07-26T06:20:20.418 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64407.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64407.log.gz 2024-07-26T06:20:20.419 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64491.log 2024-07-26T06:20:20.420 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.55208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62429.log 2024-07-26T06:20:20.420 INFO:teuthology.orchestra.run.smithi163.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55208.log.gz 2024-07-26T06:20:20.421 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64491.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.64491.log.gz 2024-07-26T06:20:20.421 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44011.log 2024-07-26T06:20:20.422 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.62429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64365.log 2024-07-26T06:20:20.422 INFO:teuthology.orchestra.run.smithi163.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.62429.log.gz 2024-07-26T06:20:20.424 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.44011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48452.log 2024-07-26T06:20:20.424 INFO:teuthology.orchestra.run.smithi163.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44011.log.gz 2024-07-26T06:20:20.425 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.64365.log: gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64365.log.gz 2024-07-26T06:20:20.425 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50068.log 2024-07-26T06:20:20.426 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.48452.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48452.log.gz 2024-07-26T06:20:20.426 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.50068.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.50068.log.gz 2024-07-26T06:20:20.446 INFO:teuthology.orchestra.run.smithi163.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-07-26T06:20:20.521 INFO:teuthology.orchestra.run.smithi163.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-07-26T06:20:20.536 INFO:teuthology.orchestra.run.smithi163.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-07-26T06:20:20.604 INFO:teuthology.orchestra.run.smithi163.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-07-26T06:20:20.660 INFO:teuthology.orchestra.run.smithi163.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-07-26T06:20:20.664 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-07-26T06:20:20.665 INFO:teuthology.orchestra.run.smithi163.stderr:real 0m0.980s 2024-07-26T06:20:20.665 INFO:teuthology.orchestra.run.smithi163.stderr:user 0m3.579s 2024-07-26T06:20:20.665 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m1.489s 2024-07-26T06:20:20.666 INFO:tasks.ceph:Archiving logs... 2024-07-26T06:20:20.666 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/remote/smithi163/log 2024-07-26T06:20:20.667 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-07-26T06:20:21.436 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/remote/smithi164/log 2024-07-26T06:20:21.437 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-07-26T06:20:21.851 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-07-26T06:20:21.864 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-07-26T06:20:21.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-07-26T06:20:21.872 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-07-26T06:20:22.000 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-07-26T06:20:22.009 INFO:teuthology.task.clock:Checking final clock skew... 2024-07-26T06:20:22.010 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:20:22.017 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:20:22.065 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:20:22.072 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:20:22.075 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:20:22.075 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-07-26T06:20:22.075 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +952us[ +953us] +/- 62ms 2024-07-26T06:20:22.075 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -1103us[-1102us] +/- 63ms 2024-07-26T06:20:22.075 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +318us[ +318us] +/- 88ms 2024-07-26T06:20:22.075 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:20:22.081 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:20:22.081 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-07-26T06:20:22.081 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 +988us[ +998us] +/- 62ms 2024-07-26T06:20:22.081 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -1161us[-1151us] +/- 63ms 2024-07-26T06:20:22.082 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +331us[ +331us] +/- 88ms 2024-07-26T06:20:22.082 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:20:22.083 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-07-26T06:20:22.094 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-07-26T06:20:22.095 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-07-26T06:20:22.109 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-07-26T06:20:22.293 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-07-26T06:20:22.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:20:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:20:22.585 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:20:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:20:22.678 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-07-26T06:20:22.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-07-26T06:20:22.703 INFO:teuthology.task.internal:Duration was 1014.674855 seconds 2024-07-26T06:20:22.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-07-26T06:20:22.718 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-07-26T06:20:22.719 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-07-26T06:20:22.728 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-07-26T06:20:22.857 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:20:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:20:23.350 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-07-26T06:20:23.350 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:20:23.351 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-07-26T06:20:23.414 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:20:23.414 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-07-26T06:20:23.490 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-07-26T06:20:23.491 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-07-26T06:20:23.499 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-07-26T06:20:23.584 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-07-26T06:20:23.585 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-07-26T06:20:23.591 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-07-26T06:20:23.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-07-26T06:20:23.806 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-07-26T06:20:23.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-07-26T06:20:23.892 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-07-26T06:20:23.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-07-26T06:20:23.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-07-26T06:20:24.000 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-07-26T06:20:24.094 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-07-26T06:20:24.098 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-07-26T06:20:24.139 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-07-26T06:20:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:20:24.187 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-07-26T06:20:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:20:24.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-07-26T06:20:24.253 INFO:teuthology.task.internal:Transferring archived files... 2024-07-26T06:20:24.253 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/remote/smithi163 2024-07-26T06:20:24.254 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-07-26T06:20:24.350 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-07-26_05:39:42-fs:volumes-rishabh-mgr-vol-clone-stats-bugfixes-testing-default-smithi/7818960/remote/smithi164 2024-07-26T06:20:24.350 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-07-26T06:20:24.446 INFO:teuthology.task.internal:Removing archive directory... 2024-07-26T06:20:24.447 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-07-26T06:20:24.456 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-07-26T06:20:24.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-07-26T06:20:24.525 INFO:teuthology.task.internal:Not uploading archives. 2024-07-26T06:20:24.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-07-26T06:20:24.540 INFO:teuthology.task.internal:Tidying up after the test... 2024-07-26T06:20:24.541 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-07-26T06:20:24.548 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-07-26T06:20:24.596 INFO:teuthology.orchestra.run.smithi163.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 26 06:20 /home/ubuntu/cephtest 2024-07-26T06:20:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 26 06:20 /home/ubuntu/cephtest 2024-07-26T06:20:24.608 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-07-26T06:20:24.622 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-07-26T06:20:24.657 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3e9255e907100f88049f320ca33d977e29155d12 client.1-kernel-sha1: 3e9255e907100f88049f320ca33d977e29155d12 description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} duration: 1014.6748549938202 failure_reason: 'Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d1bfbf22bbe944c8a893c46e866aba68 status: fail success: false 2024-07-26T06:20:24.657 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T06:20:24.739 INFO:teuthology.run:FAIL